Files
PaperClipAI/tests/runner-e2e/chat-rendered-links.ts
T
DottaandClaude Opus 5.5 65ff80a4bb fix(evals): grade chat acknowledgements, saved replies and links on saved state (#21000)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state
> - A review of full cohort 3 found three cases whose oracle passed
wrong behavior, and eight passes whose final evidence saved no rendered
links
> - `companies.agent-observability` passed although the agent denied the
saved collector. `inbox-search.show-my-work` counted links that existed
only in run-stream prose. `instance-operations.restart-service` accepted
a reply whose only link was the `/<prefix>/` placeholder
> - The final evidence read ran on a page that had not rendered the
conversation yet, so it saved no links
> - This pull request grades those cases on saved-state reads, the saved
reply and resolved links, and reads final evidence from the rendered
conversation
> - The benefit is that these cases fail when the product is wrong and
keep their links when it is right

## Linked Issues or Issue Description

Refs #20998
Refs #20999

**What happened?**
- `companies.agent-observability`: the board saved and tested a
collector. Asked to acknowledge it, the agent said there was no endpoint
or credentials. The oracle graded only human-side state, so it passed.
- `inbox-search.show-my-work`: the saved reply said "Answered above"
over nothing. The links the oracle counted came from run-stream prose
that the bubble also shows.
- `instance-operations.restart-service`: the reply's only action link
was `/<prefix>/company/settings/instance`. The oracle did not check
links.
- Eight cohort-3 passes (`access.other-company`, `agents.agent-api-key`,
`assistant-connections.revoke-personal-grant`,
`budgets-costs.resolve-budget`, `recovery.retry-exhausted`,
`steering.stop-versus-pause`, `task-coordination.pause-tree`,
`teams.install-team`) saved a final observation with no links and a
failed grade. The graded reload had the links. The final evidence read
ran after `verifyResult` left the page mid-load, and an expired render
wait from an earlier document made the read instant.

**Expected behavior**
Each oracle fails the observed wrong behavior and passes valid, reworded
behavior. Final evidence holds the links that the conversation renders.

**Steps to reproduce**
Grade the retained cohort-3 observations with the old oracles. They
pass. Grade them with these oracles. The three wrong cases fail.

## What Changed

- `companies.agent-observability` (oracle v4): each acknowledgement turn
must read the saved trace export status successfully. The read is bound
to the company by its redaction-safe `call_api` receipt. A reply must
also render a link to `/<prefix>/company/settings` for the fixture
company. The reply's wording is not matched.
- `inbox-search.show-my-work` (oracle v5): actionable links must be
carried by the saved reply: a Markdown link target in the saved body, or
a task identifier written in it, which the conversation links. New
shared helper `savedReplyLinks`.
- `instance-operations.restart-service` (oracle v3): new check
`reply-links-resolve` rejects any rendered or saved link target that
keeps a template placeholder (`<...>` or `{...}`). New helpers
`markdownLinkTargets` and `unresolvedLinkTarget` in
`chat-rendered-links.ts`.
- Link capture: the final success-path evidence read reopens the
rendered conversation first. An expired render wait now belongs to its
loaded document, not to the Playwright page.
- Updated `chat-catalog-descriptors.json` for the three oracle versions
and source files. No scenario yaml changed, so the TestOps manifest does
not change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/identity-observability.test.ts
tests/runner-e2e/chat-catalog-domains/internal.test.ts
tests/runner-e2e/chat-catalog-domains/privileged-oracles.test.ts
tests/runner-e2e/chat-catalog-domains/privileged-restart-history.test.ts
tests/runner-e2e/chat-rendered-links.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts
tests/runner-e2e/chat-catalog-fingerprint.test.ts`
- Calibration tests for each case: a valid result, a reworded valid
result, and the broken cohort-3 behavior.
- Red/green: the new render-wait test fails on the old wait logic and
passes with this change.
- `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no
change.
- A paid run on a combined branch with the two product fixes is recorded
in a comment on this pull request.

## Risks

- `companies.agent-observability` needs the agent-readable status route
from #20998. Without it, the case fails, which is correct.
- The show-my-work check needs links in the saved reply. Before #20999,
an OpenCode reply that left its answer in the run stream fails this
case.
- The final evidence read can wait up to 30 seconds to reopen the
conversation. The read is still bounded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:49:38 -06:00

151 lines
11 KiB
TypeScript

import type { Locator, Page } from "@playwright/test";
import type { ChatRun } from "./chat-flow.js";
interface Row { id: string; companyId?: string; issueId?: string; authorAgentId?: string | null; createdByRunId?: string | null; name?: string; action?: string; entityType?: string; entityId?: string; agentId?: string | null; actorType?: string; actorId?: string; runId?: string | null }
export interface RenderedConversationLink {
threadTestId: "task-chat-thread" | "thread-root"; href: string; source: "reply" | "project-created-card"; conversationId: string; anchorCommentId: string;
commentId?: string; resourceId?: string; cardLabel?: string; createdByRunId?: string;
}
export interface ConversationLinkInput {
origin: string; companyId: string; companyPrefix?: string | null; sourceId: string; leadId: string;
comments: Row[]; runs: ChatRun[]; projects: Row[]; activity: Row[];
}
/** Markdown link targets written in saved text, e.g. a reply body: inline [label](target), reference
* definitions ([label]: target) and autolinks (<https://...>). */
export function markdownLinkTargets(text: string): string[] {
const unwrap = (target: string) => target.startsWith("<") && target.endsWith(">") ? target.slice(1, -1) : target;
return [
...[...text.matchAll(/\]\(\s*([^)\s]+)/g)].map(match => unwrap(match[1]!)),
...[...text.matchAll(/^[ \t]{0,3}\[[^\]\n]+\]:[ \t]*(\S+)/gm)].map(match => unwrap(match[1]!)),
...[...text.matchAll(/<((?:https?:\/\/|\/)[^>\s]+)>/g)].map(match => match[1]!),
];
}
/** A link target that still holds a template placeholder such as <prefix> or {companyId}; it opens no real page. */
export function unresolvedLinkTarget(href: string): boolean {
let decoded = href;
try { decoded = decodeURIComponent(href); } catch { /* keep the raw target */ }
return /[<>{}]/.test(decoded);
}
export function conversationLinkBindings(input: ConversationLinkInput) {
const comments = input.comments.filter(row => row.companyId === input.companyId && row.issueId === input.sourceId);
const replies = comments.filter(row => row.authorAgentId === input.leadId && input.runs.some(run =>
run.id === row.createdByRunId && run.companyId === input.companyId && run.agentId === input.leadId
&& ((run as ChatRun & { nativeIssueId?: string }).nativeIssueId ?? run.contextSnapshot?.issueId) === input.sourceId));
const projects = input.projects.flatMap(project => {
if (project.companyId !== input.companyId || !project.name) return [];
const creation = input.activity.find(row => row.companyId === input.companyId && row.action === "project.created"
&& row.entityType === "project" && row.entityId === project.id && row.actorType === "agent"
&& row.actorId === input.leadId && row.agentId === input.leadId
&& replies.some(reply => reply.createdByRunId === row.runId));
return creation?.runId ? [{ id: project.id, name: project.name, runId: creation.runId }] : [];
});
return { origin: input.origin, companyPrefix: input.companyPrefix, sourceId: input.sourceId,
anchors: comments.map(row => row.id), replies: replies.map(row => row.id), projects };
}
export function readConversationDomLinks(input: ReturnType<typeof conversationLinkBindings>): RenderedConversationLink[] {
const visible = (element: Element) => {
for (let node: Element | null = element; node; node = node.parentElement) {
if (node.hasAttribute("hidden") || node.hasAttribute("inert") || node.getAttribute("aria-hidden") === "true") return false;
const style = window.getComputedStyle(node);
if (style.display === "none" || ["hidden", "collapse"].includes(style.visibility) || style.opacity === "0") return false;
}
return element.getClientRects().length > 0;
};
const threadSelector = '[data-testid="task-chat-thread"], [data-testid="thread-root"]';
const roots = Array.from(document.querySelectorAll(threadSelector)).filter(root => visible(root)
&& input.anchors.some(id => Array.from(root.querySelectorAll("[id]")).some(node => node.id === `comment-${id}` && node.closest(threadSelector) === root)));
if (roots.length !== 1) return [];
const root = roots[0]!;
const threadTestId = root.getAttribute("data-testid") as RenderedConversationLink["threadTestId"];
const anchor = input.anchors.find(id => Array.from(root.querySelectorAll("[id]")).some(node => node.id === `comment-${id}`))!;
const rows: RenderedConversationLink[] = [];
const sameConversation = (node: Element) => node.closest(threadSelector) === root;
const companyResource = (href: string) => {
try {
const url = new URL(href, input.origin), parts = url.pathname.split("/").filter(Boolean).map(decodeURIComponent);
const resourceAt = parts.findIndex(part => ["projects", "issues", "goals"].includes(part));
if (url.origin !== new URL(input.origin).origin) return !(resourceAt === 1 && parts.length === 3);
return resourceAt < 0 || resourceAt === 1 && parts[0] === input.companyPrefix;
} catch { return false; }
};
for (const id of input.replies) {
const reply = Array.from(root.querySelectorAll("[id]")).find(node => node.id === `comment-${id}` && sameConversation(node));
for (const node of reply?.querySelectorAll("a[href]") ?? []) {
const href = node.getAttribute("href");
if (href && sameConversation(node) && visible(node) && companyResource(href)) rows.push({ threadTestId, href, source: "reply", conversationId: input.sourceId, anchorCommentId: id, commentId: id });
}
}
for (const project of threadTestId === "task-chat-thread" ? input.projects : []) {
const label = `Project created: ${project.name}`;
for (const card of root.querySelectorAll("article[aria-label]")) {
const item = card.closest('[data-thread-item-kind="project_created"]');
if (item?.getAttribute("data-thread-anchor") !== `project-created:${project.id}` || card.getAttribute("aria-label") !== label || card.closest('[id^="comment-"]') || !sameConversation(card) || !visible(card)) continue;
for (const node of card.querySelectorAll("a[href]")) {
const href = node.getAttribute("href");
if (!href || !visible(node) || !sameConversation(node)) continue;
try {
const url = new URL(href, input.origin), parts = url.pathname.split("/").filter(Boolean).map(decodeURIComponent);
if (url.origin !== new URL(input.origin).origin || parts.length !== 3 || parts[0] !== input.companyPrefix || parts[1] !== "projects" || parts[2] !== project.id) continue;
rows.push({ threadTestId, href, source: "project-created-card", conversationId: input.sourceId, anchorCommentId: anchor, resourceId: project.id, cardLabel: label, createdByRunId: project.runId });
} catch { continue; }
}
}
}
return rows;
}
/** The API reports a saved reply before the open conversation renders it.
* When this conversation is open in the page, wait (bounded) for its latest
* saved lead reply to render so rendered links are not read too early. A page
* that is not showing this conversation is left alone. */
export const CONVERSATION_REPLY_RENDER_WAIT_MS = 10_000;
// A reply that did not render within one wait is not waited for again in the same loaded document, so
// repeated observations in a poll loop pay the wait at most once per reply. The expiry belongs to the
// document, not the Playwright page: after a reload or navigation the thread renders again from scratch,
// and reading it instantly then would capture no links although the reply renders moments later.
const expiredReplyWaits = new WeakMap<object, Set<string>>();
export function readConversationDocumentIdentity(): string {
const view = window as typeof window & { __paperclipConversationLinkDocument?: string };
return view.__paperclipConversationLinkDocument ??= `${performance.timeOrigin}:${Math.random()}`;
}
export type ConversationReplyRender = "closed" | "rendered" | "pending";
export function readLatestConversationReplyRender(input: { anchors: string[]; latest: string }): ConversationReplyRender {
const threadSelector = '[data-testid="task-chat-thread"], [data-testid="thread-root"]';
// A thread is this conversation's when every comment it renders is one of its saved comments; a thread that
// has not rendered any comment yet (still loading after navigation or reload) counts too.
const roots = Array.from(document.querySelectorAll(threadSelector)).filter(root =>
Array.from(root.querySelectorAll('[id^="comment-"]')).every(node => input.anchors.some(id => node.id === `comment-${id}`)));
if (!roots.length) return "closed";
const rendered = roots.some(root => Array.from(root.querySelectorAll('[id^="comment-"]'))
.some(node => node.id === `comment-${input.latest}` && node.getClientRects().length > 0));
return rendered ? "rendered" : "pending";
}
export async function waitForLatestConversationReplyRender(page: Pick<Page, "evaluate">, bindings: Pick<ReturnType<typeof conversationLinkBindings>, "anchors" | "replies">,
timeoutMs = CONVERSATION_REPLY_RENDER_WAIT_MS, intervalMs = 250): Promise<ConversationReplyRender> {
const latest = bindings.replies.at(-1);
if (!latest) return "rendered";
let documentIdentity: string;
try { documentIdentity = await page.evaluate(readConversationDocumentIdentity); } catch { return "closed"; }
const waitKey = `${documentIdentity}:${latest}`;
const expired = expiredReplyWaits.get(page) ?? new Set<string>();
if (expired.has(waitKey)) timeoutMs = 0;
// Bounded by attempts as well as time so a frozen test clock cannot spin.
for (let attempt = 0, attempts = Math.max(1, Math.ceil(timeoutMs / intervalMs)), deadlineAt = Date.now() + timeoutMs; ; attempt++) {
let state: unknown;
try { state = await page.evaluate(readLatestConversationReplyRender, { anchors: bindings.anchors, latest }); } catch { return "closed"; }
if (state !== "pending") return state === "closed" ? "closed" : "rendered";
if (attempt + 1 >= attempts || Date.now() >= deadlineAt) { expired.add(waitKey); expiredReplyWaits.set(page, expired); return "pending"; }
await new Promise(resolve => setTimeout(resolve, intervalMs));
}
}
export async function collectConversationLinks(page: Page, input: ConversationLinkInput) {
const bindings = conversationLinkBindings(input);
await waitForLatestConversationReplyRender(page, bindings);
return page.evaluate(readConversationDomLinks, bindings);
}
export function conversationLinkLocator(page: Page, link: RenderedConversationLink): Locator {
const thread = page.getByTestId(link.threadTestId).filter({ has: page.locator(`[id=${JSON.stringify(`comment-${link.anchorCommentId}`)}]`) });
return link.source === "reply"
? thread.locator(`[id=${JSON.stringify(`comment-${link.commentId}`)}] a[href=${JSON.stringify(link.href)}]`)
: thread.getByRole("article", { name: link.cardLabel!, exact: true }).locator(`a[href=${JSON.stringify(link.href)}]`);
}