mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 08:53:19 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog checks that a chat agent does the work a person asks for > - In the latest full-cohort run on master, 16 cases failed although the saved work was correct > - Their graders matched words, one literal value, or one API route in free-form model output > - A grader that fails correct work hides real failures in the noise, and it can also pass wrong work that uses the expected word > - This pull request grades those cases by saved state, audit records, setup literals and rendered page structure, and keeps only narrow checks for false claims > - The benefit is that these cases fail only when the product or agent does the wrong thing ## Linked Issues or Issue Description Refs #18971 (related: company-secret question handoffs) **What happened** The full-cohort chat-eval run on master failed 16 cases where the retained evidence shows correct behaviour. Examples: `companies.rename-company` accepted only `company.updated`, but the supported branding route logs `company.branding_updated`. `recovery.post-stop-message` required a signup button label that matched a regex, and rejected a labelled email form with the button "Start writing free". `secrets.secret-proposal` flagged "the value is delivered when Morgan next runs" as a false delivery claim. Other graders required words such as "uncommitted", "telemetry", "administrator" or the app name that the question did not ask for. **Expected behavior** A grader accepts every supported route and wording that achieves the outcome. It fails when saved state is wrong, when a requested fact is wrong, or when the reply makes a claim that saved state contradicts. **Steps to reproduce** 1. Load the retained observation for one of the cases below from the full-cohort evidence. 2. Grade it with the old oracle. The keyword or route check fails although the saved state is correct. 3. Grade it with the new oracle. It passes. A broken variant (wrong saved state or a contradicting claim) still fails. ## What Changed - `activity.audit-change`: attribution is graded by structure. The reply names the recorded editor, names no other company agent (new setup literal `otherActors`), and states only real transitions. Sentence grammar is no longer matched. - `app-permissions.ask-before-action`: each continuation reply must exist and must link no app record other than the reviewed connection. It no longer must name the app. - `budgets-costs.spend-breakdown`: the reply must point at the record with unresolved usage (task link, identifier or run id) and must not give it a zero or free amount. The words "subscription" and "incomplete" are no longer required. - `cases.browse-cases`: an excluded case may be mentioned in prose. It must not be linked or listed as a result row. - `cases.handoff-same-task`: handoff instructions carry findings forward when they point at the saved `findings` document, reproduce it, or restate it. The premise (a primary Morgan run performs the handoff) is unchanged. - `cases.update-case-facts`: a saved browser value may add clauses that only rule out other platforms ("desktop Safari unaffected"). - `companies.agent-observability`: the reply no longer must name Telemetry and the run log. It must not claim that they are exported. - `companies.rename-company`: both supported name routes (`company.updated`, `company.branding_updated`) are accepted. - `connection-setup.choose-provider`: the answer must exist and must not claim a connected account. The unavailable method no longer must be named; the setup path stays graded by clicking the card or link. - `goals.align-existing-work`: a question card from the answering run counts as the phase-two response. Refusal words and "unchanged" words are no longer required; false change claims still fail. - `recovery.post-stop-message` (and `steering.interrupt-with-direction`, which shares the helper): a visible form with a labelled field and a submit control is a signup action, whatever the button says. - `routines.pause-owned`: the "paused"/"running" words are no longer required. The saved checkpoint proves both; a cancellation claim still fails. - `secrets.secret-proposal` (and `secrets.configure-company-secret`, which shares the helper): a sentence that says when delivery will happen is not a delivery claim. - `skills.explicit-restriction`: the reply no longer must say "administrator". Success claims and invented tool limitations still fail. - `workspaces.inspect-git`: the reply no longer must say "uncommitted". Unchanged HEAD and status prove it; commit or push claims still fail. - Every affected `oracleVersion` is bumped in code and in `chat-catalog-descriptors.json`. No scenario YAML changed. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. - `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact canary with Chromium: pass, including new form cases. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass. `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged pins. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Offline calibration on retained evidence, for each case: the retained correct evidence passes, earlier passing evidence passes where it exists, and a broken variant fails. `spend-breakdown` retained evidence still fails, because its reply shows unresolved usage as `$0.00`. `handoff-same-task` retained evidence still fails on the kept primary-run premise. - Live paid runs: see the comment below. ## Risks - Low risk. The change is limited to eval graders and their unit tests. Product code does not change. - Each grader is more accepting of wording. Saved-state checks and the narrow false-claim checks remain, so a wrong outcome still fails. - `secrets.configure-company-secret` still fails its retained evidence. Its agent asked a two-question card, and the act supports only one-question cards. This pull request changes it only by the shared delivery-claim helper. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
76 lines
6.8 KiB
TypeScript
76 lines
6.8 KiB
TypeScript
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
export interface RuntimeRenderedNode { tag: string; role: string | null; text: string; href: string | null; inNavigation: boolean; color: string; background: string;
|
|
/** FORM nodes only: a visible labelled email field and a visible submit control inside the form. */
|
|
form?: { labelledField: boolean; submit: boolean } }
|
|
/** Render only independently downloaded artifact bytes, with no scripts, network or authenticated storage. */
|
|
export async function renderRuntimeArtifact(ctx: ChatCatalogContext, html: string): Promise<RuntimeRenderedNode[]> {
|
|
if (Buffer.byteLength(html) > 262_144) throw new Error("Rendered runtime artifact exceeds its registered bounded size");
|
|
const browser = ctx.input.page.context().browser(); if (!browser) throw new Error("Actual rendered artifact inspection requires the existing browser");
|
|
const preview = await browser.newContext({ javaScriptEnabled: false });
|
|
try {
|
|
const page = await preview.newPage(); await page.route("**/*", route => route.abort());
|
|
await page.setContent(html, { waitUntil: "domcontentloaded" });
|
|
return await page.locator("body, main, section, header, h1, h2, h3, p, button, a, nav, form, [role]").evaluateAll(elements => {
|
|
if (elements.length > 2_000) throw new Error("Rendered artifact exceeds its finite DOM inspection bound");
|
|
const visible = (element: Element) => {
|
|
const rect = element.getBoundingClientRect(); if (rect.width < 2 || rect.height < 2 || rect.right <= 0 || rect.left >= innerWidth) return false;
|
|
for (let node: Element | null = element; node; node = node.parentElement) { const style = getComputedStyle(node);
|
|
if (style.display === "none" || style.visibility === "hidden" || Number(style.opacity) === 0) return false; }
|
|
return true;
|
|
};
|
|
const text = (element: Element) => {
|
|
const walker = document.createTreeWalker(element, NodeFilter.SHOW_TEXT), parts: string[] = []; let node: Node | null; let count = 0;
|
|
while ((node = walker.nextNode())) { if (++count > 10_000) throw new Error("Rendered artifact exceeds its finite text inspection bound");
|
|
if (node.parentElement && visible(node.parentElement) && node.textContent?.trim()) parts.push(node.textContent.trim()); }
|
|
return parts.join(" ").replace(/\s+/g, " ").slice(0, 20_000);
|
|
};
|
|
const formShape = (form: Element) => {
|
|
const fields = [...form.querySelectorAll("input, textarea")].filter(field => visible(field)
|
|
&& !["hidden", "submit", "button", "reset", "checkbox", "radio", "image"].includes((field.getAttribute("type") ?? "text").toLowerCase()));
|
|
// A real accessible name: aria-label text, an existing aria-labelledby target with text, or a
|
|
// wrapping/for label with text. A signup field takes an email address.
|
|
const labelled = (field: Element) => !!field.getAttribute("aria-label")?.trim()
|
|
|| (field.getAttribute("aria-labelledby") ?? "").split(/\s+/).filter(Boolean).some(id => !!form.ownerDocument.getElementById(id)?.textContent?.trim())
|
|
|| !!field.closest("label")?.textContent?.trim()
|
|
|| (!!field.id && [...form.ownerDocument.querySelectorAll("label")].some(label => label.getAttribute("for") === field.id && !!label.textContent?.trim()));
|
|
// An email entry: type or autocomplete email, or a text field whose own label, name or placeholder says
|
|
// email. A search field (type search, or labelled only "Search") is not a signup field.
|
|
const accessibleName = (field: Element) => [field.getAttribute("aria-label"), field.closest("label")?.textContent,
|
|
...(field.getAttribute("aria-labelledby") ?? "").split(/\s+/).filter(Boolean).map(id => form.ownerDocument.getElementById(id)?.textContent),
|
|
...(field.id ? [...form.ownerDocument.querySelectorAll("label")].filter(label => label.getAttribute("for") === field.id).map(label => label.textContent) : [])].join(" ");
|
|
const emailField = (field: Element) => { const type = (field.getAttribute("type") ?? "text").toLowerCase();
|
|
return type === "email" || /\bemail\b/i.test(field.getAttribute("autocomplete") ?? "")
|
|
|| (type === "text" && /\be-?mail\b/i.test([accessibleName(field), field.getAttribute("name"), field.getAttribute("placeholder")].join(" "))); };
|
|
const submit = [...form.querySelectorAll("button, input[type=submit], input[type=image]")].some(control => visible(control)
|
|
&& !["button", "reset"].includes((control.getAttribute("type") ?? "submit").toLowerCase()));
|
|
return { labelledField: fields.some(field => labelled(field) && emailField(field)), submit };
|
|
};
|
|
return elements.filter(visible).map(element => { const style = getComputedStyle(element);
|
|
return { tag: element.tagName, role: element.getAttribute("role"), text: text(element) || element.getAttribute("aria-label") || "", href: element.getAttribute("href"),
|
|
inNavigation: !!element.closest('nav, [role="navigation"]'), color: style.color, background: style.backgroundColor,
|
|
...(element.tagName === "FORM" ? { form: formShape(element) } : {}) }; });
|
|
});
|
|
} finally { await preview.close(); }
|
|
}
|
|
/** Closed sub-results of the single-page check, for diagnosis without retaining page text. */
|
|
export function simplerWebsiteChecks(nodes: RuntimeRenderedNode[]) {
|
|
// A visible heading, and the product named somewhere visible. The request asked for a heading,
|
|
// not for the brand inside it (a tagline H1 under a brand header is a natural design).
|
|
const headline = nodes.some(node => (node.tag === "H1" || node.role === "heading") && node.text.trim().length > 0)
|
|
&& nodes.some(node => /\bAcorn\s+Notes\b/i.test(node.text));
|
|
// A visible product paragraph; the request never named a <main> landmark.
|
|
const content = nodes.some(node => ["MAIN", "SECTION", "P"].includes(node.tag) && node.text.length >= 40);
|
|
// A labelled signup action: structurally, a visible form with a labelled email field and a submit
|
|
// control, whatever its button says ("Start writing free" is as valid as "Sign up"); or a visible
|
|
// control that is itself labelled as a signup action.
|
|
const signup = nodes.some(node => node.tag === "FORM" && node.form?.labelledField === true && node.form.submit === true)
|
|
|| nodes.some(node => ["BUTTON", "A"].includes(node.tag) && /sign.?up|subscribe|join|early access/i.test(node.text));
|
|
const tabbed = nodes.some(node => ["tab", "tablist", "tabpanel"].includes(node.role ?? ""));
|
|
const multiPage = nodes.some(node => node.tag === "A" && node.inNavigation && !!node.href && !/^#|^\s*$/.test(node.href));
|
|
return { headline, content, signup, tabbed, multiPage };
|
|
}
|
|
export function hasSimplerWebsite(nodes: RuntimeRenderedNode[]) {
|
|
const { headline, content, signup, tabbed, multiPage } = simplerWebsiteChecks(nodes);
|
|
return headline && content && signup && !tabbed && !multiPage;
|
|
}
|