Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/internal-support.ts
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

138 lines
9.5 KiB
TypeScript

import { collectConversationLinks, conversationLinkLocator, type RenderedConversationLink } from "../chat-rendered-links.js";
import { expect } from "@playwright/test";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import type { ChatRun } from "../chat-flow.js";
export interface Entity {
id: string; companyId?: string; name?: string; title?: string; description?: string | null;
identifier?: string; urlKey?: string; status?: string; priority?: string; assigneeAgentId?: string | null;
projectId?: string | null; parentId?: string | null; goalId?: string | null; goalIds?: string[];
level?: string; ownerAgentId?: string | null; archivedAt?: string | null;
key?: string; body?: string; latestRevisionId?: string; latestRevisionNumber?: number;
createdByAgentId?: string | null; updatedByAgentId?: string | null; createdByRunId?: string | null;
createdAt?: string; updatedAt?: string; authorAgentId?: string | null; authorUserId?: string | null;
issueId?: string; action?: string; agentId?: string | null; actorId?: string; actorType?: string;
entityId?: string; documentId?: string; originalRevisionId?: string | null; currentRevisionId?: string | null; resolvedByAgentId?: string | null; selectedText?: string; resolvedAt?: string | null; details?: Record<string, unknown>;
}
export interface InternalObservation {
scenarioId: string; origin: string; companyId: string; leadId: string; state: Record<string, unknown>;
tasks: Entity[]; projects: Entity[]; goals: Entity[]; comments: Entity[];
renderedLinkEvidence?: RenderedConversationLink[]; activity: Entity[]; annotations?: Entity[]; agents?: Entity[]; documents: Entity[]; revisions: Entity[]; reply: string; links: string[]; humanQueries: string[]; runs: ChatRun[];
}
export function entity(state: Record<string, unknown>, key: string): Entity {
const value = state[key];
if (!value || typeof value !== "object" || typeof (value as Entity).id !== "string") throw new Error(`Missing fixture ${key}`);
return value as Entity;
}
export const companyPath = (ctx: ChatCatalogContext) => `/api/companies/${ctx.input.fixtures.company.id}`;
export async function project(ctx: ChatCatalogContext, name = "Launch", extra: Record<string, unknown> = {}) {
return ctx.input.api.post<Entity>(`${companyPath(ctx)}/projects`, { name, ...extra });
}
export async function task(ctx: ChatCatalogContext, title: string, projectId: string, extra: Record<string, unknown> = {}) {
// Backlog fixtures never start provider work. These are prerequisite records, not requested outcomes.
return ctx.input.api.post<Entity>(`${companyPath(ctx)}/issues`, { title, projectId, status: "backlog", priority: "medium", ...extra });
}
export async function goal(ctx: ChatCatalogContext, title: string, extra: Record<string, unknown> = {}) {
return ctx.input.api.post<Entity>(`${companyPath(ctx)}/goals`, { title, level: "company", ownerAgentId: ctx.input.fixtures.agent.id, ...extra });
}
export async function putDocument(ctx: ChatCatalogContext, issueId: string, body: string, baseRevisionId: string | null = null) {
return ctx.input.api.putSensitive<Entity>(`/api/issues/${issueId}/documents/plan`, {
title: "Website implementation plan", format: "markdown", body, baseRevisionId,
});
}
export async function observeInternal(ctx: ChatCatalogContext, state: Record<string, unknown>): Promise<InternalObservation> {
await ctx.refreshIssue();
const { api, fixtures } = ctx.input;
const [tasks, projects, goals, runs] = await Promise.all([
api.getRunObservation<Entity[]>(`${companyPath(ctx)}/issues?includeAgentCreatedTasks=true&limit=1000`),
api.get<Entity[]>(`${companyPath(ctx)}/projects?includeArchived=true`),
api.get<Entity[]>(`${companyPath(ctx)}/goals`), ctx.allRuns(),
]);
const source = ctx.issue();
const chat = source ? await api.get<Entity[]>(`/api/issues/${source.id}/comments?order=asc`) : [];
const replies = chat.filter(row => row.authorAgentId === fixtures.agent.id && runs.some(run => run.id === row.createdByRunId));
const target = state.task ? entity(state, "task") : undefined;
const comments = target ? await api.get<Entity[]>(`/api/issues/${target.id}/comments?order=asc`) : [];
const companyActivity = await api.getRunObservation<Entity[]>(`${companyPath(ctx)}/activity?limit=1000`);
const taskActivity = target ? await api.get<Entity[]>(`/api/issues/${target.id}/activity`) : [];
const activity = [...new Map([...companyActivity, ...taskActivity].map(row => [row.id, row])).values()];
const renderedLinkEvidence = source ? await collectConversationLinks(ctx.input.page, {
origin: api.baseURL, companyId: fixtures.company.id, companyPrefix: fixtures.company.issuePrefix,
sourceId: source.id, leadId: fixtures.agent.id, comments: chat, runs, projects, activity,
}) : [];
const links = [...new Set(renderedLinkEvidence.map(link => link.href))];
const documents: Entity[] = []; const revisions: Entity[] = [];
if (target) {
for (const summary of await api.get<Entity[]>(`/api/issues/${target.id}/documents`)) {
const endpoint = `/api/issues/${target.id}/documents/${encodeURIComponent(summary.key!)}`;
documents.push(await api.get<Entity>(endpoint));
revisions.push(...await api.get<Entity[]>(`${endpoint}/revisions`));
}
}
return { scenarioId: ctx.scenario.id, origin: api.baseURL, companyId: fixtures.company.id, leadId: fixtures.agent.id,
state, tasks: tasks.filter(row => row.id !== source?.id), projects, goals, runs, comments, activity, documents, revisions,
renderedLinkEvidence, humanQueries: chat.filter(row => !row.authorAgentId).map(row => row.body ?? ""), reply: replies.map(row => row.body ?? "").join("\n"), links };
}
export function hasLink(o: InternalObservation, kind: "issues" | "projects" | "goals", row: Entity): boolean {
return o.links.some(href => {
try {
const url = new URL(href, o.origin);
if (url.origin !== new URL(o.origin).origin) return false;
const parts = url.pathname.split("/").map(decodeURIComponent); const index = parts.indexOf(kind);
const target = parts[index + 1];
return index >= 0 && !!target && [row.id, row.identifier, row.urlKey].filter(Boolean).includes(target);
} catch { return false; }
});
}
export function observation(value: unknown): InternalObservation | undefined {
if (!value || typeof value !== "object") return;
const o = value as InternalObservation;
if (!o.state || ![o.tasks, o.projects, o.goals, o.comments, o.activity, o.documents, o.revisions, o.links, o.humanQueries, o.runs].every(Array.isArray)
|| typeof o.reply !== "string" || typeof o.origin !== "string" || typeof o.companyId !== "string" || typeof o.leadId !== "string") return;
return o;
}
export function checks() {
const rows: Array<{ id: string; passed: boolean; detail: string }> = [];
return { add(id: string, passed: boolean, detail: string) { rows.push({ id, passed, detail }); },
result() { return { passed: rows.length > 0 && rows.every(row => row.passed), checks: rows }; } };
}
export function sameTask(before: Entity, after: Entity | undefined, except: readonly string[] = []) {
return !!after && ["id", "companyId", "title", "description", "projectId", "status", "priority", "assigneeAgentId", "goalId"]
.filter(key => !except.includes(key)).every(key => before[key as keyof Entity] === after[key as keyof Entity]);
}
/** Follow the exact rendered chat links, check the real result view, and return to the source. */
export async function verifyInternalResult(ctx: ChatCatalogContext, _state: Record<string, unknown>, value: unknown, remainingBudget?: () => number) {
const options = () => remainingBudget ? { timeout: remainingBudget() } : {};
const o = observation(value);
if (!o) throw new Error("Result navigation requires a complete observation");
const sourceUrl = ctx.input.page.url();
const candidates: Array<{ kind: "issues" | "projects" | "goals"; row: Entity }> = [
...o.tasks.map(row => ({ kind: "issues" as const, row })),
...o.projects.map(row => ({ kind: "projects" as const, row })),
...o.goals.map(row => ({ kind: "goals" as const, row })),
].filter(target => hasLink(o, target.kind, target.row));
if (!candidates.length) throw new Error("No independently matched result link was rendered");
for (const { kind, row } of candidates) {
const href = o.links.find(link => hasLink({ ...o, links: [link] }, kind, row))!;
const link = internalResultLink(ctx, o, href).first();
await expect(link).toBeVisible(options());
await link.click(options());
await ctx.input.page.waitForURL(url => {
const parts = url.pathname.split("/").map(decodeURIComponent); const at = parts.indexOf(kind);
return at >= 0 && [row.id, row.identifier, row.urlKey].includes(parts[at + 1]);
}, options());
await expect(ctx.input.page.getByRole("heading", { name: row.title ?? row.name!, exact: true }).first()).toBeVisible(options());
await ctx.input.evidence(`chat-catalog-opened-${kind}-${row.id}.json`, { id: row.id, kind, url: ctx.input.page.url(), title: row.title ?? row.name });
await ctx.input.page.goto(sourceUrl, { waitUntil: "domcontentloaded", ...options() });
await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible(options());
remainingBudget?.();
}
}
export function internalResultLink(ctx: ChatCatalogContext, o: InternalObservation, href: string) {
const receipt = o.renderedLinkEvidence?.find(link => link.href === href);
if (!receipt) throw new Error("Result navigation requires current-conversation rendered-link custody");
return conversationLinkLocator(ctx.input.page, receipt);
}