mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog runs real agent conversations and grades the saved outcome > - Chat-eval cohort 16 (OpenCode with a DeepSeek model) had three failures that were not product failures of the chat agent > - One setup check compared time zone names, one observer read a form before it filled, and one grader accepted only exact query digests > - An eval that fails valid behavior hides real regressions behind noise, so each check must grade the outcome, not an incidental detail > - This pull request fixes the three checks, and it fixes a first-frame empty display name on the Profile page > - The benefit is that these three cases now fail only for the behavior they test ## Linked Issues or Issue Description **What happened?** Three chat-eval cases failed in cohort 16 (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`) on correct behavior. (1) `activity.routine-audit`: setup threw "Browser timezone override did not apply (America/Buenos_Aires)". At 11:00-11:59 UTC the picker selects `America/Argentina/Buenos_Aires`, and Chrome reports that zone by its alias `America/Buenos_Aires`. The check compared names. (2) `navigation-preferences.edit-profile`: the name saved correctly, but the observer read the display-name field once, right after reload, and got an empty value. (3) `budgets-costs.spend-breakdown`: the answer was correct, but the grader accepted company cost reports only for `period=all` or `period=month`. The agent read `costs/by-agent` and `costs/summary` for the requested week, and that range covers the unpriced run. **Expected behavior** Each case passes when the product and the agent do the requested work. Each case fails when they do not. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.activity.routine-audit` between 11:00 and 11:59 UTC. 2. Run `chat-features.runner-opencode.local.navigation-preferences.edit-profile` and look at `profileAfterReload`. 3. Grade the retained cohort 16 observation of `budgets-costs.spend-breakdown` with the old grader. **Additional context** The run tool inputs are saved only as SHA-256 digests, so a grader binds a read by recomputing the digest of inputs it can check. For the cohort 16 spend-breakdown reads, the inputs were recovered this way: `from=2026-11-09T00:00:00.000Z`, `to=2026-11-16T00:00:00.000Z`. ## What Changed - `internal-routine-audit.ts`: the browser override check compares the UTC offset of the requested and reported zones, now and 182 days later, instead of the zone names. An unknown zone or a different clock still fails setup. - `identity-profile.ts`: after reload, `settledFieldValue` waits up to 15 s for the display name to reach the saved value, then records what the person sees. The grade still fails a name that does not arrive. - `ProfileSettings.tsx`: unedited fields show the session's saved name and image from the first frame. Before, a session restored from the saved read cache rendered an empty name until an effect filled it. - `identity-accounting.ts`: `readSavedAccounting` also accepts a company cost report whose checked `from`/`to` covers the unpriced run's cost event. The run log keeps only the input digest, so the grader recomputes digests for covering ranges: UTC day boundaries (with or without milliseconds or seconds, date-only, end of day), local midnights at each whole-hour offset, and the run's own `startedAt` and cost-event instants. A pending run (no cost event) counts only through `costs/summary`, `costs/data-quality` or its own run record, because the other reports sum cost events. A link to the run does not count, because the run id is in the task description. The run-log redaction is not changed. - Oracle versions bumped: `routine-audit.v7`, `spend-breakdown.structural.v6`, `identity-profile.v4`. ## Verification - Red/green: each new test fails on the old code and passes on the new code. - `internal-remaining.test.ts`: every candidate zone, reported by its own name, its ICU name and known browser aliases, passes at every hour in January and July. A different clock, a zone with daylight saving time and an unknown name fail. - `identity-personal.test.ts`: the settled read returns the saved name after empty reads, and returns a wrong name when the saved name never arrives. - `ProfileSettings.test.tsx`: the first rendered frame with a cached session shows "Alex Chen", not `""`. - `identity-accounting.test.ts`: the cohort 16 week-range reads pass, as do other covering ranges. Negative controls fail: no reads, run-list reads with unchecked inputs, ranges that exclude the run, another company, a denied read, a read by another run, and a read of a different run record with a link to the target run. - The retained cohort 16 spend-breakdown observation passes the new grader, and fails with its reads removed. - `pnpm test:e2e:runner:unit`: 686 files passed. `pnpm test:e2e:runner:typecheck` passed. `ui` ProfileSettings tests passed. `scripts/check-token-gates.mjs` is clean. - One narrow paid run of the three cases on this branch (result in a comment below). ## Risks - Low risk. The changes are in eval code, plus a small Profile page state change. - A covering range in a form outside the recomputed set (for example 08:00 at -03:00) is not recognized. That is a false negative, not a false pass. - The Profile page now keeps an unedited field in step with the session. A refetch still resets drafts, as before. ## Model Used - Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
328 lines
11 KiB
TypeScript
328 lines
11 KiB
TypeScript
// @vitest-environment jsdom
|
|
|
|
import { act } from "react";
|
|
import { createRoot } from "react-dom/client";
|
|
import { renderToStaticMarkup } from "react-dom/server";
|
|
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { sanitizeAssetNamespace } from "@paperclipai/shared";
|
|
import { ApiError } from "@/api/client";
|
|
import { queryKeys } from "../lib/queryKeys";
|
|
import { markCachedSession } from "../lib/cached-session";
|
|
import { startPersistedQueryCache } from "../lib/persisted-query-cache";
|
|
import { ProfileSettings } from "./ProfileSettings";
|
|
|
|
const mockAuthApi = vi.hoisted(() => ({
|
|
getSession: vi.fn(),
|
|
signInEmail: vi.fn(),
|
|
signUpEmail: vi.fn(),
|
|
getProfile: vi.fn(),
|
|
updateProfile: vi.fn(),
|
|
signOut: vi.fn(),
|
|
}));
|
|
|
|
const mockAssetsApi = vi.hoisted(() => ({
|
|
uploadImage: vi.fn(),
|
|
uploadCompanyLogo: vi.fn(),
|
|
}));
|
|
|
|
const mockSetBreadcrumbs = vi.hoisted(() => vi.fn());
|
|
|
|
vi.mock("@/api/auth", () => ({
|
|
authApi: mockAuthApi,
|
|
}));
|
|
|
|
vi.mock("@/components/InboxAgentPolicyControl", () => ({ InboxAgentPolicyControl: () => null }));
|
|
|
|
vi.mock("@/api/assets", () => ({
|
|
assetsApi: mockAssetsApi,
|
|
}));
|
|
|
|
vi.mock("../context/BreadcrumbContext", () => ({
|
|
useBreadcrumbs: () => ({
|
|
setBreadcrumbs: mockSetBreadcrumbs,
|
|
}),
|
|
}));
|
|
|
|
vi.mock("../context/CompanyContext", () => ({
|
|
useCompany: () => ({
|
|
selectedCompanyId: "company-1",
|
|
selectedCompany: { id: "company-1", name: "Paperclip", issuePrefix: "PAP" },
|
|
}),
|
|
}));
|
|
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
(globalThis as any).IS_REACT_ACT_ENVIRONMENT = true;
|
|
|
|
async function flushReact() {
|
|
await act(async () => {
|
|
await Promise.resolve();
|
|
await new Promise((resolve) => window.setTimeout(resolve, 0));
|
|
});
|
|
}
|
|
|
|
describe("ProfileSettings", () => {
|
|
let container: HTMLDivElement;
|
|
|
|
beforeEach(() => {
|
|
container = document.createElement("div");
|
|
document.body.appendChild(container);
|
|
|
|
mockAuthApi.getSession.mockResolvedValue({
|
|
session: { id: "session-1", userId: "user-1" },
|
|
user: {
|
|
id: "user-1",
|
|
name: "Jane Example",
|
|
email: "jane@example.com",
|
|
image: "https://example.com/jane.png",
|
|
},
|
|
});
|
|
mockAssetsApi.uploadImage.mockResolvedValue({
|
|
assetId: "asset-1",
|
|
contentPath: "/api/assets/asset-1/content",
|
|
});
|
|
mockAuthApi.updateProfile.mockImplementation(async (input: { name: string; image: string | null }) => ({
|
|
id: "user-1",
|
|
name: input.name,
|
|
email: "jane@example.com",
|
|
image: input.image,
|
|
}));
|
|
});
|
|
|
|
afterEach(() => {
|
|
container.remove();
|
|
document.body.innerHTML = "";
|
|
vi.clearAllMocks();
|
|
});
|
|
|
|
it("shows the saved display name in the first frame when the session is already cached", () => {
|
|
// A reload restores the session from the saved read cache before any response. The first
|
|
// committed frame must show the saved name, not an empty field that an effect fills later.
|
|
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity } } });
|
|
queryClient.setQueryData(queryKeys.auth.session, {
|
|
session: { id: "session-1", userId: "user-1" },
|
|
user: { id: "user-1", name: "Alex Chen", email: "alex@example.com", image: null },
|
|
});
|
|
const html = renderToStaticMarkup(<QueryClientProvider client={queryClient}><ProfileSettings /></QueryClientProvider>);
|
|
const field = new DOMParser().parseFromString(html, "text/html").getElementById("profile-name") as HTMLInputElement | null;
|
|
expect(field?.getAttribute("value")).toBe("Alex Chen");
|
|
});
|
|
|
|
it("does not render a keyboard shortcuts toggle because shortcuts are always enabled", async () => {
|
|
const root = createRoot(container);
|
|
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } });
|
|
await act(async () => {
|
|
root.render(<QueryClientProvider client={queryClient}><ProfileSettings /></QueryClientProvider>);
|
|
});
|
|
await flushReact();
|
|
expect(container.textContent).toContain("Jane Example");
|
|
expect(container.textContent).not.toContain("Keyboard shortcuts");
|
|
expect(container.querySelector('[aria-label="Toggle keyboard shortcuts"]')).toBeNull();
|
|
await act(async () => root.unmount());
|
|
});
|
|
|
|
it("uploads a clicked avatar into Paperclip storage and persists the returned asset path", async () => {
|
|
const root = createRoot(container);
|
|
const queryClient = new QueryClient({
|
|
defaultOptions: { queries: { retry: false } },
|
|
});
|
|
|
|
await act(async () => {
|
|
root.render(
|
|
<QueryClientProvider client={queryClient}>
|
|
<ProfileSettings />
|
|
</QueryClientProvider>,
|
|
);
|
|
});
|
|
await flushReact();
|
|
await flushReact();
|
|
|
|
expect(container.textContent).not.toContain("Avatar image URL");
|
|
|
|
const avatarInput = container.querySelector('input[type="file"]') as HTMLInputElement | null;
|
|
expect(avatarInput).not.toBeNull();
|
|
|
|
const file = new File(["avatar"], "avatar.png", { type: "image/png" });
|
|
Object.defineProperty(avatarInput, "files", {
|
|
configurable: true,
|
|
value: [file],
|
|
});
|
|
|
|
await act(async () => {
|
|
avatarInput?.dispatchEvent(new Event("change", { bubbles: true }));
|
|
});
|
|
await flushReact();
|
|
await flushReact();
|
|
|
|
expect(mockAssetsApi.uploadImage).toHaveBeenCalledWith("company-1", file, "profiles/user-1");
|
|
expect(mockAuthApi.updateProfile).toHaveBeenCalledWith({
|
|
name: "Jane Example",
|
|
image: "/api/assets/asset-1/content",
|
|
});
|
|
|
|
await act(async () => {
|
|
root.unmount();
|
|
});
|
|
});
|
|
|
|
it("uploads an avatar for a user id that comes from an identity provider", async () => {
|
|
const userId = "oidc:example|jane.example@example.com";
|
|
mockAuthApi.getSession.mockResolvedValue({
|
|
session: { id: "session-1", userId },
|
|
user: {
|
|
id: userId,
|
|
name: "Jane Example",
|
|
email: "jane@example.com",
|
|
image: "https://example.com/jane.png",
|
|
},
|
|
});
|
|
|
|
const root = createRoot(container);
|
|
const queryClient = new QueryClient({
|
|
defaultOptions: { queries: { retry: false } },
|
|
});
|
|
|
|
await act(async () => {
|
|
root.render(
|
|
<QueryClientProvider client={queryClient}>
|
|
<ProfileSettings />
|
|
</QueryClientProvider>,
|
|
);
|
|
});
|
|
await flushReact();
|
|
await flushReact();
|
|
|
|
const avatarInput = container.querySelector('input[type="file"]') as HTMLInputElement | null;
|
|
expect(avatarInput).not.toBeNull();
|
|
|
|
const file = new File(["avatar"], "avatar.png", { type: "image/png" });
|
|
Object.defineProperty(avatarInput, "files", {
|
|
configurable: true,
|
|
value: [file],
|
|
});
|
|
|
|
await act(async () => {
|
|
avatarInput?.dispatchEvent(new Event("change", { bubbles: true }));
|
|
});
|
|
await flushReact();
|
|
await flushReact();
|
|
|
|
const namespace = `profiles/${userId}`;
|
|
expect(mockAssetsApi.uploadImage).toHaveBeenCalledWith("company-1", file, namespace);
|
|
// The namespace goes to the API without a change, so the avatar arrives
|
|
// under the identity of the user.
|
|
expect(sanitizeAssetNamespace(namespace)).toBe(namespace);
|
|
expect(mockAuthApi.updateProfile).toHaveBeenCalledWith({
|
|
name: "Jane Example",
|
|
image: "/api/assets/asset-1/content",
|
|
});
|
|
|
|
await act(async () => {
|
|
root.unmount();
|
|
});
|
|
});
|
|
|
|
async function renderPage(queryClient: QueryClient) {
|
|
const root = createRoot(container);
|
|
await act(async () => {
|
|
root.render(
|
|
<QueryClientProvider client={queryClient}>
|
|
<ProfileSettings />
|
|
</QueryClientProvider>,
|
|
);
|
|
});
|
|
await flushReact();
|
|
await flushReact();
|
|
return root;
|
|
}
|
|
|
|
it("keeps the loaded profile on screen when a session refetch fails transiently", async () => {
|
|
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } });
|
|
const root = await renderPage(queryClient);
|
|
expect(container.textContent).toContain("Jane Example");
|
|
|
|
mockAuthApi.getSession.mockRejectedValue(
|
|
new ApiError("Paperclip is restarting.", 503, { error: "tenant_app_unavailable" }),
|
|
);
|
|
await act(async () => {
|
|
await queryClient.invalidateQueries({ queryKey: queryKeys.auth.session });
|
|
});
|
|
await flushReact();
|
|
|
|
expect(mockAuthApi.getSession).toHaveBeenCalledTimes(2);
|
|
expect(container.textContent).toContain("Jane Example");
|
|
expect(container.querySelector('[data-query-view="error"]')).toBeNull();
|
|
expect(container.textContent).not.toContain("Paperclip is restarting.");
|
|
expect(container.textContent).not.toContain("tenant_app_unavailable");
|
|
|
|
await act(async () => root.unmount());
|
|
});
|
|
|
|
it("shows readable copy and a Retry button when the session fails to load", async () => {
|
|
mockAuthApi.getSession.mockRejectedValueOnce(new ApiError("Boom", 500, { error: "Boom" }));
|
|
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } });
|
|
const root = await renderPage(queryClient);
|
|
|
|
expect(container.textContent).toContain("Couldn't load your profile");
|
|
expect(container.querySelector('[data-query-view="error"]')).not.toBeNull();
|
|
expect(container.textContent).not.toContain("Jane Example");
|
|
const retryButton = Array.from(container.querySelectorAll("button")).find(
|
|
(button) => button.textContent === "Retry",
|
|
);
|
|
expect(retryButton).toBeTruthy();
|
|
|
|
await act(async () => {
|
|
retryButton?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
|
});
|
|
await flushReact();
|
|
await flushReact();
|
|
|
|
expect(container.textContent).toContain("Jane Example");
|
|
expect(container.querySelector('[data-query-view="error"]')).toBeNull();
|
|
|
|
await act(async () => root.unmount());
|
|
});
|
|
|
|
it.each(["user-1", "user-2"])("waits for session verification before saving a restored profile (%s)", async (verifiedUser) => {
|
|
const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity }, mutations: { retry: false } } });
|
|
queryClient.setQueryData(queryKeys.auth.session, {
|
|
session: { id: "session-1", userId: "user-1" },
|
|
user: { id: "user-1", name: "Jane Example", email: "jane@example.com", image: null },
|
|
});
|
|
const cache = startPersistedQueryCache(queryClient, { storage: null, buildId: "test", eventTarget: null });
|
|
markCachedSession(queryClient);
|
|
let complete!: (value: unknown) => void;
|
|
mockAuthApi.getSession.mockReturnValue(new Promise((resolve) => { complete = resolve; }));
|
|
const root = await renderPage(queryClient);
|
|
try {
|
|
expect(container.textContent).toContain("Jane Example");
|
|
await act(async () => {
|
|
container.querySelector("form")?.dispatchEvent(new Event("submit", { bubbles: true, cancelable: true }));
|
|
});
|
|
await flushReact();
|
|
expect(mockAuthApi.getSession).toHaveBeenCalled();
|
|
expect(mockAuthApi.updateProfile).not.toHaveBeenCalled();
|
|
|
|
await act(async () => {
|
|
complete({
|
|
session: { id: "session-2", userId: verifiedUser },
|
|
user: { id: verifiedUser, name: "Verified User", email: "verified@example.com", image: null },
|
|
});
|
|
});
|
|
await flushReact();
|
|
await flushReact();
|
|
|
|
if (verifiedUser === "user-1") {
|
|
expect(mockAuthApi.updateProfile).toHaveBeenCalledTimes(1);
|
|
expect(mockAuthApi.updateProfile).toHaveBeenCalledWith({ name: "Jane Example", image: null });
|
|
} else {
|
|
expect(mockAuthApi.updateProfile).not.toHaveBeenCalled();
|
|
}
|
|
} finally {
|
|
await act(async () => root.unmount());
|
|
cache.dispose();
|
|
queryClient.clear();
|
|
}
|
|
});
|
|
});
|