Files
PaperClipAI/ui/src/components/task-chat/QuestionForm.test.tsx
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

135 lines
8.5 KiB
TypeScript

// @vitest-environment jsdom
import { act } from "react";
import { createRoot, type Root } from "react-dom/client";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { PaperclipQuestionResponse, PaperclipQuestionSet } from "@paperclipai/adapter-utils";
import { QuestionForm } from "./QuestionForm";
vi.mock("./TaskChatRichInput", () => ({
TaskChatRichInput: ({ value, onChange, onSubmit }: { value: string; onChange(value: string): void; onSubmit(): void }) =>
<textarea aria-label="Answer" value={value} onChange={event => onChange(event.target.value)} onKeyDown={event => { if (event.key === "Enter" && event.metaKey) onSubmit(); }} />,
}));
vi.mock("@/components/MarkdownBody", () => ({ MarkdownBody: ({ children }: { children: string }) => <div>{children}</div> }));
const questionSet: PaperclipQuestionSet = { schema: "paperclip.question_set.v1", questions: [
{ id: "draft", prompt: "Edit the draft", required: true, answerMode: "text", initialText: " Draft\n漢字\n" },
] };
describe("QuestionForm initial text", () => {
let container: HTMLDivElement;
let root: Root;
const submit = vi.fn();
beforeEach(() => { localStorage.clear(); submit.mockReset(); container = document.createElement("div"); document.body.append(container); root = createRoot(container); });
afterEach(async () => { await act(async () => root.unmount()); container.remove(); localStorage.clear(); });
const input = () => container.querySelector("textarea")!;
async function render(set = questionSet, initialResponse?: PaperclipQuestionResponse) {
await act(async () => root.render(<QuestionForm id="edit" questionSet={set} initialResponse={initialResponse} draftKey="question-initial-text" onSubmit={submit} />));
}
async function edit(text: string) {
await act(async () => {
Object.getOwnPropertyDescriptor(HTMLTextAreaElement.prototype, "value")!.set!.call(input(), text);
input().dispatchEvent(new Event("input", { bubbles: true }));
});
}
async function remount(set = questionSet) { await act(async () => root.unmount()); root = createRoot(container); await render(set); }
it("shows exact editable text without sending until the operator submits", async () => {
await render();
expect(input().value).toBe(" Draft\n漢字\n");
expect(submit).not.toHaveBeenCalled();
await act(async () => Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Submit answers")!.click());
expect(submit).toHaveBeenCalledExactlyOnceWith({ schema: "paperclip.question_response.v1", answers: { draft: { text: " Draft\n漢字\n" } } });
});
it("preserves edits through a refreshed request and reload instead of reapplying a default", async () => {
await render(); await edit("Operator edit\n");
const refreshed = { ...questionSet, questions: [{ ...questionSet.questions[0]!, initialText: "New provider default" }] };
await render(refreshed); expect(input().value).toBe("Operator edit\n");
await remount(refreshed); expect(input().value).toBe("Operator edit\n");
expect(submit).not.toHaveBeenCalled();
});
it("shows required validation without advancing or submitting after enabled Next", async () => {
await render({ schema: "paperclip.question_set.v1", questions: [
{ id: "audience", prompt: "Who is the audience?", required: true, answerMode: "text" },
{ id: "tone", prompt: "What tone should we use?", required: true, answerMode: "text" },
] });
const next = Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Next")!;
expect(next.disabled).toBe(false);
await act(async () => next.click());
expect(container.textContent).toContain("This question is required.");
expect(container.textContent).toContain("Who is the audience?");
expect(container.textContent).not.toContain("What tone should we use?");
expect(submit).not.toHaveBeenCalled();
});
it("retains an explicitly cleared draft and required validation across reload", async () => {
await render(); await edit(""); await remount();
expect(input().value).toBe("");
expect(container.querySelector('[role="alert"]')).toBeNull();
await act(async () => Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Submit answers")!.click());
expect(container.textContent).toContain("This question is required.");
expect(submit).not.toHaveBeenCalled();
});
it("isolates pending requests with the same field ids and restores each edited draft when switching back", async () => {
async function showRequest(id: string, initialText: string) {
const set = { ...questionSet, questions: [{ ...questionSet.questions[0]!, initialText }] };
await act(async () => root.render(
<QuestionForm id={id} questionSet={set} draftKey={`question:${id}`} onSubmit={submit} />,
));
}
const storedText = (id: string) => JSON.parse(localStorage.getItem(`question:${id}`)!).answers.draft.text;
await showRequest("request-a", "Provider draft A");
await edit("Operator edit A");
await showRequest("request-b", "Provider draft B");
expect(input().value).toBe("Provider draft B");
expect(storedText("request-a")).toBe("Operator edit A");
expect(storedText("request-b")).toBe("Provider draft B");
await edit("Operator edit B");
await showRequest("request-a", "Refreshed provider draft A");
expect(input().value).toBe("Operator edit A");
await showRequest("request-b", "Refreshed provider draft B");
expect(input().value).toBe("Operator edit B");
expect(storedText("request-a")).toBe("Operator edit A");
expect(storedText("request-b")).toBe("Operator edit B");
expect(submit).not.toHaveBeenCalled();
await act(async () => Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Submit answers")!.click());
expect(submit).toHaveBeenCalledExactlyOnceWith({ schema: "paperclip.question_response.v1", answers: { draft: { text: "Operator edit B" } } });
});
it("prefers an existing response and keeps text constraints on defaults", async () => {
await render(questionSet, { schema: "paperclip.question_response.v1", answers: { draft: { text: "Existing answer" } } });
expect(input().value).toBe("Existing answer");
await act(async () => root.unmount()); localStorage.clear(); root = createRoot(container);
await render({ ...questionSet, questions: [{ ...questionSet.questions[0]!, initialText: "bad", textValidation: { minLength: 5 } }] });
expect(input().value).toBe("bad");
expect(container.textContent).not.toContain("Enter at least 5 characters.");
await act(async () => Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Submit answers")!.click());
expect(container.textContent).toContain("Enter at least 5 characters.");
expect(submit).not.toHaveBeenCalled();
});
it("only validates a short answer after Next or keyboard submit, and clears it when corrected", async () => {
await render({ schema: "paperclip.question_set.v1", questions: [
{ id: "audience", prompt: "Who is it for?", required: true, answerMode: "text", textValidation: { minLength: 2 } },
{ id: "outcome", prompt: "What outcome?", required: true, answerMode: "text", textValidation: { minLength: 2 } },
] });
await edit("m");
expect(container.textContent).not.toContain("Enter at least 2 characters.");
await act(async () => Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Next")!.click());
expect(container.textContent).toContain("Enter at least 2 characters.");
expect(container.textContent).toContain("Who is it for?");
await edit("millennials");
expect(container.querySelector('[role="alert"]')).toBeNull();
await act(async () => Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Next")!.click());
await edit("x");
expect(container.querySelector('[role="alert"]')).toBeNull();
await act(async () => input().dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", metaKey: true, bubbles: true })));
expect(container.textContent).toContain("Enter at least 2 characters.");
expect(submit).not.toHaveBeenCalled();
await edit("10 concepts");
await act(async () => Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Submit answers")!.click());
expect(submit).toHaveBeenCalledExactlyOnceWith({ schema: "paperclip.question_response.v1", answers: {
audience: { text: "millennials" }, outcome: { text: "10 concepts" },
} });
});
});