Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/runtime-rendered-artifact.ts
T
DottaandClaude Opus 5.5 5f34f777ec fix(evals): grade 16 chat cases by saved state instead of reply keywords (#20600)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog checks that a chat agent does the work
a person asks for
> - In the latest full-cohort run on master, 16 cases failed although
the saved work was correct
> - Their graders matched words, one literal value, or one API route in
free-form model output
> - A grader that fails correct work hides real failures in the noise,
and it can also pass wrong work that uses the expected word
> - This pull request grades those cases by saved state, audit records,
setup literals and rendered page structure, and keeps only narrow checks
for false claims
> - The benefit is that these cases fail only when the product or agent
does the wrong thing

## Linked Issues or Issue Description

Refs #18971 (related: company-secret question handoffs)

**What happened**

The full-cohort chat-eval run on master failed 16 cases where the
retained evidence shows correct behaviour. Examples:
`companies.rename-company` accepted only `company.updated`, but the
supported branding route logs `company.branding_updated`.
`recovery.post-stop-message` required a signup button label that matched
a regex, and rejected a labelled email form with the button "Start
writing free". `secrets.secret-proposal` flagged "the value is delivered
when Morgan next runs" as a false delivery claim. Other graders required
words such as "uncommitted", "telemetry", "administrator" or the app
name that the question did not ask for.

**Expected behavior**

A grader accepts every supported route and wording that achieves the
outcome. It fails when saved state is wrong, when a requested fact is
wrong, or when the reply makes a claim that saved state contradicts.

**Steps to reproduce**

1. Load the retained observation for one of the cases below from the
full-cohort evidence.
2. Grade it with the old oracle. The keyword or route check fails
although the saved state is correct.
3. Grade it with the new oracle. It passes. A broken variant (wrong
saved state or a contradicting claim) still fails.

## What Changed

- `activity.audit-change`: attribution is graded by structure. The reply
names the recorded editor, names no other company agent (new setup
literal `otherActors`), and states only real transitions. Sentence
grammar is no longer matched.
- `app-permissions.ask-before-action`: each continuation reply must
exist and must link no app record other than the reviewed connection. It
no longer must name the app.
- `budgets-costs.spend-breakdown`: the reply must point at the record
with unresolved usage (task link, identifier or run id) and must not
give it a zero or free amount. The words "subscription" and "incomplete"
are no longer required.
- `cases.browse-cases`: an excluded case may be mentioned in prose. It
must not be linked or listed as a result row.
- `cases.handoff-same-task`: handoff instructions carry findings forward
when they point at the saved `findings` document, reproduce it, or
restate it. The premise (a primary Morgan run performs the handoff) is
unchanged.
- `cases.update-case-facts`: a saved browser value may add clauses that
only rule out other platforms ("desktop Safari unaffected").
- `companies.agent-observability`: the reply no longer must name
Telemetry and the run log. It must not claim that they are exported.
- `companies.rename-company`: both supported name routes
(`company.updated`, `company.branding_updated`) are accepted.
- `connection-setup.choose-provider`: the answer must exist and must not
claim a connected account. The unavailable method no longer must be
named; the setup path stays graded by clicking the card or link.
- `goals.align-existing-work`: a question card from the answering run
counts as the phase-two response. Refusal words and "unchanged" words
are no longer required; false change claims still fail.
- `recovery.post-stop-message` (and `steering.interrupt-with-direction`,
which shares the helper): a visible form with a labelled field and a
submit control is a signup action, whatever the button says.
- `routines.pause-owned`: the "paused"/"running" words are no longer
required. The saved checkpoint proves both; a cancellation claim still
fails.
- `secrets.secret-proposal` (and `secrets.configure-company-secret`,
which shares the helper): a sentence that says when delivery will happen
is not a delivery claim.
- `skills.explicit-restriction`: the reply no longer must say
"administrator". Success claims and invented tool limitations still
fail.
- `workspaces.inspect-git`: the reply no longer must say "uncommitted".
Unchanged HEAD and status prove it; commit or push claims still fail.
- Every affected `oracleVersion` is bumped in code and in
`chat-catalog-descriptors.json`. No scenario YAML changed.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000 tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass.
- `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact
canary with Chromium: pass, including new form cases.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged pins.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- Offline calibration on retained evidence, for each case: the retained
correct evidence passes, earlier passing evidence passes where it
exists, and a broken variant fails. `spend-breakdown` retained evidence
still fails, because its reply shows unresolved usage as `$0.00`.
`handoff-same-task` retained evidence still fails on the kept
primary-run premise.
- Live paid runs: see the comment below.

## Risks

- Low risk. The change is limited to eval graders and their unit tests.
Product code does not change.
- Each grader is more accepting of wording. Saved-state checks and the
narrow false-claim checks remain, so a wrong outcome still fails.
- `secrets.configure-company-secret` still fails its retained evidence.
Its agent asked a two-question card, and the act supports only
one-question cards. This pull request changes it only by the shared
delivery-claim helper.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:37:16 -06:00

76 lines
6.8 KiB
TypeScript

import type { ChatCatalogContext } from "../chat-catalog-contract.js";
export interface RuntimeRenderedNode { tag: string; role: string | null; text: string; href: string | null; inNavigation: boolean; color: string; background: string;
/** FORM nodes only: a visible labelled email field and a visible submit control inside the form. */
form?: { labelledField: boolean; submit: boolean } }
/** Render only independently downloaded artifact bytes, with no scripts, network or authenticated storage. */
export async function renderRuntimeArtifact(ctx: ChatCatalogContext, html: string): Promise<RuntimeRenderedNode[]> {
if (Buffer.byteLength(html) > 262_144) throw new Error("Rendered runtime artifact exceeds its registered bounded size");
const browser = ctx.input.page.context().browser(); if (!browser) throw new Error("Actual rendered artifact inspection requires the existing browser");
const preview = await browser.newContext({ javaScriptEnabled: false });
try {
const page = await preview.newPage(); await page.route("**/*", route => route.abort());
await page.setContent(html, { waitUntil: "domcontentloaded" });
return await page.locator("body, main, section, header, h1, h2, h3, p, button, a, nav, form, [role]").evaluateAll(elements => {
if (elements.length > 2_000) throw new Error("Rendered artifact exceeds its finite DOM inspection bound");
const visible = (element: Element) => {
const rect = element.getBoundingClientRect(); if (rect.width < 2 || rect.height < 2 || rect.right <= 0 || rect.left >= innerWidth) return false;
for (let node: Element | null = element; node; node = node.parentElement) { const style = getComputedStyle(node);
if (style.display === "none" || style.visibility === "hidden" || Number(style.opacity) === 0) return false; }
return true;
};
const text = (element: Element) => {
const walker = document.createTreeWalker(element, NodeFilter.SHOW_TEXT), parts: string[] = []; let node: Node | null; let count = 0;
while ((node = walker.nextNode())) { if (++count > 10_000) throw new Error("Rendered artifact exceeds its finite text inspection bound");
if (node.parentElement && visible(node.parentElement) && node.textContent?.trim()) parts.push(node.textContent.trim()); }
return parts.join(" ").replace(/\s+/g, " ").slice(0, 20_000);
};
const formShape = (form: Element) => {
const fields = [...form.querySelectorAll("input, textarea")].filter(field => visible(field)
&& !["hidden", "submit", "button", "reset", "checkbox", "radio", "image"].includes((field.getAttribute("type") ?? "text").toLowerCase()));
// A real accessible name: aria-label text, an existing aria-labelledby target with text, or a
// wrapping/for label with text. A signup field takes an email address.
const labelled = (field: Element) => !!field.getAttribute("aria-label")?.trim()
|| (field.getAttribute("aria-labelledby") ?? "").split(/\s+/).filter(Boolean).some(id => !!form.ownerDocument.getElementById(id)?.textContent?.trim())
|| !!field.closest("label")?.textContent?.trim()
|| (!!field.id && [...form.ownerDocument.querySelectorAll("label")].some(label => label.getAttribute("for") === field.id && !!label.textContent?.trim()));
// An email entry: type or autocomplete email, or a text field whose own label, name or placeholder says
// email. A search field (type search, or labelled only "Search") is not a signup field.
const accessibleName = (field: Element) => [field.getAttribute("aria-label"), field.closest("label")?.textContent,
...(field.getAttribute("aria-labelledby") ?? "").split(/\s+/).filter(Boolean).map(id => form.ownerDocument.getElementById(id)?.textContent),
...(field.id ? [...form.ownerDocument.querySelectorAll("label")].filter(label => label.getAttribute("for") === field.id).map(label => label.textContent) : [])].join(" ");
const emailField = (field: Element) => { const type = (field.getAttribute("type") ?? "text").toLowerCase();
return type === "email" || /\bemail\b/i.test(field.getAttribute("autocomplete") ?? "")
|| (type === "text" && /\be-?mail\b/i.test([accessibleName(field), field.getAttribute("name"), field.getAttribute("placeholder")].join(" "))); };
const submit = [...form.querySelectorAll("button, input[type=submit], input[type=image]")].some(control => visible(control)
&& !["button", "reset"].includes((control.getAttribute("type") ?? "submit").toLowerCase()));
return { labelledField: fields.some(field => labelled(field) && emailField(field)), submit };
};
return elements.filter(visible).map(element => { const style = getComputedStyle(element);
return { tag: element.tagName, role: element.getAttribute("role"), text: text(element) || element.getAttribute("aria-label") || "", href: element.getAttribute("href"),
inNavigation: !!element.closest('nav, [role="navigation"]'), color: style.color, background: style.backgroundColor,
...(element.tagName === "FORM" ? { form: formShape(element) } : {}) }; });
});
} finally { await preview.close(); }
}
/** Closed sub-results of the single-page check, for diagnosis without retaining page text. */
export function simplerWebsiteChecks(nodes: RuntimeRenderedNode[]) {
// A visible heading, and the product named somewhere visible. The request asked for a heading,
// not for the brand inside it (a tagline H1 under a brand header is a natural design).
const headline = nodes.some(node => (node.tag === "H1" || node.role === "heading") && node.text.trim().length > 0)
&& nodes.some(node => /\bAcorn\s+Notes\b/i.test(node.text));
// A visible product paragraph; the request never named a <main> landmark.
const content = nodes.some(node => ["MAIN", "SECTION", "P"].includes(node.tag) && node.text.length >= 40);
// A labelled signup action: structurally, a visible form with a labelled email field and a submit
// control, whatever its button says ("Start writing free" is as valid as "Sign up"); or a visible
// control that is itself labelled as a signup action.
const signup = nodes.some(node => node.tag === "FORM" && node.form?.labelledField === true && node.form.submit === true)
|| nodes.some(node => ["BUTTON", "A"].includes(node.tag) && /sign.?up|subscribe|join|early access/i.test(node.text));
const tabbed = nodes.some(node => ["tab", "tablist", "tabpanel"].includes(node.role ?? ""));
const multiPage = nodes.some(node => node.tag === "A" && node.inNavigation && !!node.href && !/^#|^\s*$/.test(node.href));
return { headline, content, signup, tabbed, multiPage };
}
export function hasSimplerWebsite(nodes: RuntimeRenderedNode[]) {
const { headline, content, signup, tabbed, multiPage } = simplerWebsiteChecks(nodes);
return headline && content && signup && !tabbed && !multiPage;
}