Files
PaperClipAI/tests/runner-e2e/chat-rendered-links.ts
T
DottaandClaude Opus 5.5 cbd636d652 fix(evals): wait for rendered replies, bound runs by role and grade chat outcomes structurally (#20685)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys and grades
saved state, rendered links and run ledgers.
> - The second full-cohort run on master `9296c74e8d` failed 15 cases
where the product did the right thing and the harness did not.
> - The causes were three harness classes: links read before the reply
rendered, raw run counts that rejected product-caused follow-up runs,
and keyword graders on free-form replies. Four browser locators also
targeted retired controls.
> - This pull request fixes those harness layers. It does not change the
product.
> - The benefit is that these cases measure the feature again: correct
behaviour passes, and the relevant broken behaviour still fails.

## Linked Issues or Issue Description

**What happened**
In the second full-cohort chat-eval run on master `9296c74e8d`, these
cases failed although the reply and the saved state were correct:
`agents.choose-primary`, `cases.case-history`,
`budgets-costs.set-budget` (links read about 2s before the reply
rendered); `task-coordination.independent-review`,
`cases.update-case-facts`, `routines.manage-other-owner`,
`questions-and-approvals.review-plan` (a legitimate continuation or the
product's completion follow-up exceeded a raw run count or was
cancelled); `cases.cases-disabled`, `inbox-search.find-design`,
`navigation-preferences.broken-link`, `cases.browse-cases` (keyword or
clause graders rejected correct replies); `agent-chat.conference-room`,
`workspaces.inspect-git`, `task-coordination.block-on-design`,
`skills.author-skill` (browser steps used stale settings or controls).

**Expected behavior**
A correct reply and correct saved state pass. A wrong outcome, extra or
duplicate agent work, a substitution or a missing route still fails.

**Steps to reproduce**
Run `runner-full-stack-e2e.yml` with `cadence=manual`, `all=false` and
an `id` list of the cases above against master `9296c74e8d`. Read
`result.json` and `snapshots/` in each cell's evidence.

Refs #20600

## What Changed

- `chat-rendered-links.ts`: `collectConversationLinks` waits (at most
10s, once per reply per page) for the latest saved lead reply to render
when its conversation is open. A page that does not show the
conversation does not wait. `observeInternal` and the flow conversation
read use it.
- `budgets-costs.set-budget`: waits for the rendered reply before it
reads anchors. It fills the budget only after the form shows the saved
cap, and checks that the typed value stays.
- New `chat-catalog-domains/run-roles.ts`: identifies a run by role (a
worker woken by a comment that a named run saved on the task; the CEO's
chat completion follow-up).
- `task-coordination.independent-review`: admits one Morgan continuation
woken by the CEO's authorization comment and one Riley review run. The
review request may come from that continuation's native status decision.
Other runs still fail.
- `cases.update-case-facts`: a phase settles when its CEO turn and at
most one Morgan continuation woken by that turn's task comment finish. A
new `role-bound-runs` check rejects any other run. `terminal-counts`
gets an optional `maximumTerminalStatusCounts`.
- `routines.manage-other-owner`: admits one Morgan run on a task that
the CEO created for Morgan. A schedule change must be Morgan's own write
from that run. The ownership check is structural (a linked owner task,
or a link to the routine or Morgan's page). The keyword checks on the
reply are removed.
- `questions-and-approvals.review-plan`: after the stale-review check,
lets the product's completion follow-up settle before final run custody.
The existing grader already validates it.
- `cases.cases-disabled`: the negation-blind false-claim regex is
removed. The case passes with a reply link to the Experimental settings
page (the scripted person follows it and sees the Cases control) or a
pending choice card. Saved state still proves that nothing was created.
- `inbox-search.find-design`: the per-clause regex is removed. The first
task link in the reply must be the task with the approved revision.
- `navigation-preferences.broken-link`: a link to the similar task fails
only when its label names the original (identifier or exact title). A
separate mention passes.
- `cases.browse-cases`: an excluded case must not be a result row. When
the reply has no result list, every case link is a result, so an
excluded case must not be linked.
- Browser steps: `conference-room` reloads after the fixture enables the
flag. `inspect-git` and `block-on-design` open the panel with "Show
properties" and pick the tab from the side panel launcher.
`author-skill` does not repeat the full inspection in `verifyResult`
when `act` already inspected the same saved version.
- Each changed case has a new `oracleVersion`.
`chat-catalog-descriptors.json` is in sync.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts <119
affected test files> --testTimeout=180000`: 1673 passed, 3 skipped. One
file (`chat-factual-query-input.test.ts`) did not load in the local
worktree because `@paperclipai/plugin-sdk` is not built there. It does
not import changed code. CI runs it.
- `chat-catalog-consistency.test.ts` passes (242 descriptors).
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no pin change.
- `node --test` on the three `chat-full-catalog-testops-*.test.mjs`
files: 40 passed.
- Calibration against retained evidence: the retained replies and run
shapes of cases-disabled, browse-cases, find-design, broken-link,
independent-review, update-case-facts and manage-other-owner now pass.
Broken variants fail (closed case as a result row, earlier design linked
first, original label on the similar task's link, unprompted or
duplicate worker runs, a second review run, a routine change outside the
owner's run, no route link).
- Paid runs on this branch: see the PR comments.

## Risks

- Low risk. Only eval harness code changes.
- The structural graders do not read reply wording. A reply that words
the outcome wrongly but links the right records can pass. A person reads
the reply.
- `navigation-preferences.sidebar-order` is not changed. The shipped
sidebar has no per-project order or collapsible Agents section, so the
case premise needs an owner decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:24:41 -06:00

126 lines
9.1 KiB
TypeScript

import type { Locator, Page } from "@playwright/test";
import type { ChatRun } from "./chat-flow.js";
interface Row { id: string; companyId?: string; issueId?: string; authorAgentId?: string | null; createdByRunId?: string | null; name?: string; action?: string; entityType?: string; entityId?: string; agentId?: string | null; actorType?: string; actorId?: string; runId?: string | null }
export interface RenderedConversationLink {
threadTestId: "task-chat-thread" | "thread-root"; href: string; source: "reply" | "project-created-card"; conversationId: string; anchorCommentId: string;
commentId?: string; resourceId?: string; cardLabel?: string; createdByRunId?: string;
}
export interface ConversationLinkInput {
origin: string; companyId: string; companyPrefix?: string | null; sourceId: string; leadId: string;
comments: Row[]; runs: ChatRun[]; projects: Row[]; activity: Row[];
}
export function conversationLinkBindings(input: ConversationLinkInput) {
const comments = input.comments.filter(row => row.companyId === input.companyId && row.issueId === input.sourceId);
const replies = comments.filter(row => row.authorAgentId === input.leadId && input.runs.some(run =>
run.id === row.createdByRunId && run.companyId === input.companyId && run.agentId === input.leadId
&& ((run as ChatRun & { nativeIssueId?: string }).nativeIssueId ?? run.contextSnapshot?.issueId) === input.sourceId));
const projects = input.projects.flatMap(project => {
if (project.companyId !== input.companyId || !project.name) return [];
const creation = input.activity.find(row => row.companyId === input.companyId && row.action === "project.created"
&& row.entityType === "project" && row.entityId === project.id && row.actorType === "agent"
&& row.actorId === input.leadId && row.agentId === input.leadId
&& replies.some(reply => reply.createdByRunId === row.runId));
return creation?.runId ? [{ id: project.id, name: project.name, runId: creation.runId }] : [];
});
return { origin: input.origin, companyPrefix: input.companyPrefix, sourceId: input.sourceId,
anchors: comments.map(row => row.id), replies: replies.map(row => row.id), projects };
}
export function readConversationDomLinks(input: ReturnType<typeof conversationLinkBindings>): RenderedConversationLink[] {
const visible = (element: Element) => {
for (let node: Element | null = element; node; node = node.parentElement) {
if (node.hasAttribute("hidden") || node.hasAttribute("inert") || node.getAttribute("aria-hidden") === "true") return false;
const style = window.getComputedStyle(node);
if (style.display === "none" || ["hidden", "collapse"].includes(style.visibility) || style.opacity === "0") return false;
}
return element.getClientRects().length > 0;
};
const threadSelector = '[data-testid="task-chat-thread"], [data-testid="thread-root"]';
const roots = Array.from(document.querySelectorAll(threadSelector)).filter(root => visible(root)
&& input.anchors.some(id => Array.from(root.querySelectorAll("[id]")).some(node => node.id === `comment-${id}` && node.closest(threadSelector) === root)));
if (roots.length !== 1) return [];
const root = roots[0]!;
const threadTestId = root.getAttribute("data-testid") as RenderedConversationLink["threadTestId"];
const anchor = input.anchors.find(id => Array.from(root.querySelectorAll("[id]")).some(node => node.id === `comment-${id}`))!;
const rows: RenderedConversationLink[] = [];
const sameConversation = (node: Element) => node.closest(threadSelector) === root;
const companyResource = (href: string) => {
try {
const url = new URL(href, input.origin), parts = url.pathname.split("/").filter(Boolean).map(decodeURIComponent);
const resourceAt = parts.findIndex(part => ["projects", "issues", "goals"].includes(part));
if (url.origin !== new URL(input.origin).origin) return !(resourceAt === 1 && parts.length === 3);
return resourceAt < 0 || resourceAt === 1 && parts[0] === input.companyPrefix;
} catch { return false; }
};
for (const id of input.replies) {
const reply = Array.from(root.querySelectorAll("[id]")).find(node => node.id === `comment-${id}` && sameConversation(node));
for (const node of reply?.querySelectorAll("a[href]") ?? []) {
const href = node.getAttribute("href");
if (href && sameConversation(node) && visible(node) && companyResource(href)) rows.push({ threadTestId, href, source: "reply", conversationId: input.sourceId, anchorCommentId: id, commentId: id });
}
}
for (const project of threadTestId === "task-chat-thread" ? input.projects : []) {
const label = `Project created: ${project.name}`;
for (const card of root.querySelectorAll("article[aria-label]")) {
const item = card.closest('[data-thread-item-kind="project_created"]');
if (item?.getAttribute("data-thread-anchor") !== `project-created:${project.id}` || card.getAttribute("aria-label") !== label || card.closest('[id^="comment-"]') || !sameConversation(card) || !visible(card)) continue;
for (const node of card.querySelectorAll("a[href]")) {
const href = node.getAttribute("href");
if (!href || !visible(node) || !sameConversation(node)) continue;
try {
const url = new URL(href, input.origin), parts = url.pathname.split("/").filter(Boolean).map(decodeURIComponent);
if (url.origin !== new URL(input.origin).origin || parts.length !== 3 || parts[0] !== input.companyPrefix || parts[1] !== "projects" || parts[2] !== project.id) continue;
rows.push({ threadTestId, href, source: "project-created-card", conversationId: input.sourceId, anchorCommentId: anchor, resourceId: project.id, cardLabel: label, createdByRunId: project.runId });
} catch { continue; }
}
}
}
return rows;
}
/** The API reports a saved reply before the open conversation renders it.
* When this conversation is open in the page, wait (bounded) for its latest
* saved lead reply to render so rendered links are not read too early. A page
* that is not showing this conversation is left alone. */
export const CONVERSATION_REPLY_RENDER_WAIT_MS = 10_000;
// A reply that did not render within one wait is not waited for again on the same page, so repeated
// observations in a poll loop pay the wait at most once per reply.
const expiredReplyWaits = new WeakMap<object, Set<string>>();
export type ConversationReplyRender = "closed" | "rendered" | "pending";
export function readLatestConversationReplyRender(input: { anchors: string[]; latest: string }): ConversationReplyRender {
const threadSelector = '[data-testid="task-chat-thread"], [data-testid="thread-root"]';
// A thread is this conversation's when every comment it renders is one of its saved comments; a thread that
// has not rendered any comment yet (still loading after navigation or reload) counts too.
const roots = Array.from(document.querySelectorAll(threadSelector)).filter(root =>
Array.from(root.querySelectorAll('[id^="comment-"]')).every(node => input.anchors.some(id => node.id === `comment-${id}`)));
if (!roots.length) return "closed";
const rendered = roots.some(root => Array.from(root.querySelectorAll('[id^="comment-"]'))
.some(node => node.id === `comment-${input.latest}` && node.getClientRects().length > 0));
return rendered ? "rendered" : "pending";
}
export async function waitForLatestConversationReplyRender(page: Pick<Page, "evaluate">, bindings: Pick<ReturnType<typeof conversationLinkBindings>, "anchors" | "replies">,
timeoutMs = CONVERSATION_REPLY_RENDER_WAIT_MS, intervalMs = 250): Promise<ConversationReplyRender> {
const latest = bindings.replies.at(-1);
if (!latest) return "rendered";
const expired = expiredReplyWaits.get(page) ?? new Set<string>();
if (expired.has(latest)) timeoutMs = 0;
// Bounded by attempts as well as time so a frozen test clock cannot spin.
for (let attempt = 0, attempts = Math.max(1, Math.ceil(timeoutMs / intervalMs)), deadlineAt = Date.now() + timeoutMs; ; attempt++) {
let state: unknown;
try { state = await page.evaluate(readLatestConversationReplyRender, { anchors: bindings.anchors, latest }); } catch { return "closed"; }
if (state !== "pending") return state === "closed" ? "closed" : "rendered";
if (attempt + 1 >= attempts || Date.now() >= deadlineAt) { expired.add(latest); expiredReplyWaits.set(page, expired); return "pending"; }
await new Promise(resolve => setTimeout(resolve, intervalMs));
}
}
export async function collectConversationLinks(page: Page, input: ConversationLinkInput) {
const bindings = conversationLinkBindings(input);
await waitForLatestConversationReplyRender(page, bindings);
return page.evaluate(readConversationDomLinks, bindings);
}
export function conversationLinkLocator(page: Page, link: RenderedConversationLink): Locator {
const thread = page.getByTestId(link.threadTestId).filter({ has: page.locator(`[id=${JSON.stringify(`comment-${link.anchorCommentId}`)}]`) });
return link.source === "reply"
? thread.locator(`[id=${JSON.stringify(`comment-${link.commentId}`)}] a[href=${JSON.stringify(link.href)}]`)
: thread.getByRole("article", { name: link.cardLabel!, exact: true }).locator(`a[href=${JSON.stringify(link.href)}]`);
}