Files
PaperClipAI/tests/runner-e2e/report-navigation.ts
T
DottaandPaperclip 99d27fd5e7 feat(evals): add a metadata overview for the chat catalog (#18194)
## Thinking Path

> - Paperclip is a control plane for AI agents that do work.
> - Its evaluation reports separate recorded results from source
coverage.
> - A case without a main baseline measurement can still have a
candidate run.
> - A normal campaign card treats an expected execution without a result
as missing evidence.
> - Catalog planning data must not create those cards or change a
recorded grade.
> - This change adds a typed metadata view to the existing Evals
dashboard.
> - Readers can compare main outcomes, candidate execution, allowances
and source review separately.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The existing Evals dashboard can show a campaign's retained results. It
cannot show the full chat catalog as metadata without synthetic result
entries.

**Current behavior**

An unused initial, an unknown allowance and a completed candidate run
can all lack a main baseline result. Showing them as missing campaign
artifacts hides these differences.

**Proposed behavior**

Add one validated metadata view in the existing renderer. Show one row
per frozen scenario, separate execution and main outcome columns,
filters, and the public request and expected outcome from static YAML.
Keep normal campaign cards unchanged.

**Reason and benefit**

Readers can see which cases ran, which have no main measurement, and
which still need preparation. Source review and candidate outcomes
cannot add baseline pass credit.

**Breaking changes**

None to existing campaign inputs or outputs. The new metadata mode uses
its own strict schema and requires a new private output directory.

No matching open or closed PR or issue was found in the duplicate
search.

## What Changed

- Add a strict display-only catalog schema with closed enums and fixed
catalog scope.
- Validate the 244 row identities against static YAML without importing
executable case callbacks.
- Show main results, recorded execution, allowances and source review in
separate columns.
- Add search and filters, passing examples, qualification notes and
expandable public test descriptions.
- Add one dated immutable snapshot link in the existing shared report
navigation. This does not change history data or update an
already-published page.
- Add a private metadata preview command to the existing dashboard
owner. Keep existing campaign generation on its current path.
- Keep allowance certainty independent of measurements and allow an
optional later candidate outcome alongside a main outcome, without
changing main counters.
- Add focused pure, JSDOM and file-loader checks. Document the metadata
command.

## Verification

- The owning `catalog-overview.test.ts` Vitest suite passes all fourteen
tests under the focused source configuration. The
valid-main/unknown-allowance and main-fail/later-candidate-pass controls
fail against the old renderer and pass with this repair. The
never-run/unknown-allowance control fails against e8686bf and passes
with the restored consistency guard.
- Scoped TypeScript verification of the changed source and its imports
has no diagnostics.
- The tests cover closed schema fields, duplicate and invented IDs,
candidate/main separation, filters, HTML escaping, private output mode,
existing output refusal, symlink and oversized input refusal, static
descriptions, and the normal campaign branch. Ceiling values below,
equal to and above the target are covered. Unsupported navigation is
hidden only in metadata mode.
- The source was independently reviewed. Actual generated DOM validation
also checked 244 rows and all public descriptions.
- No visual browser verification or live product journey is claimed. No
credentials, private projection, generated report or evidence files are
included in this PR.
- Repo-wide builds were not run locally for this report-only slice.
Normal remote CI passed on repair head `f813c18`, with 55 successful
checks, two skipped checks and one neutral review check. Required `ci /
verify` and `ci / e2e` both passed. Current-head Greptile is 5/5 and no
review threads remain open. The e8686bf normal CI had an initial failed
attempt and a successful failed-only replay. Both are retained; that
replay does not substitute for current-head verification.
- The first f813c18 CI attempt passed 373 tests in serialized server
shard 4/9 but failed one existing scheduled-retry test: its intermediate
cancellation count was 29 rather than 20. The actual CI-tested test,
route, worker, scheduler and DB-fixture source matches prior passing CI.
The one failed-only replay passed all 374 tests in that shard on the
unchanged head. The first failure remains recorded; no assertion or
timeout changed.
- Rendering the authenticated existing projection through both old and
new generators produces byte-identical HTML and both assets. The
already-published dated snapshot is unchanged.

## Risks

The view depends on an authenticated metadata projection. Shape
validation does not qualify source or execution. The view preserves
unknown states and keeps candidate outcomes outside main qualification.
The preview command does not publish reports or alter history. Existing
campaign result behavior remains separate. Shared navigation gains one
dated link. Existing published pages need a separate conditional
activation; this PR does not upload or rewrite them.

## Model Used

OpenAI GPT-6 through Codex, with source editing, tool use and pure/JSDOM
checks. The exact serving model ID and context-window size are not
exposed to this worker.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 16:05:12 -05:00

179 lines
18 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { paperclipReportLogo } from "./report-brand.js";
import type { ReportExecution } from "./report-catalog.js";
import type { RunnerE2EHistoryIndex } from "./types.js";
export function reportHtml(value: unknown) {
return String(value ?? "").replace(/[&<>"']/g, character =>
({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[character]!,
);
}
export function reportHistoryHref(history: RunnerE2EHistoryIndex | undefined, campaignId: string | undefined) {
const url = history?.campaigns.find(campaign => campaign.campaignId === campaignId)?.publicUrl;
return url && /campaigns\/[^/]+\/(?:index\.html)?$/.test(url)
? url.replace(/campaigns\/[^/]+\/(?:index\.html)?$/, "")
: "https://d1p6rlowie26tp.cloudfront.net/runner-e2e/";
}
export function renderReportShell(input: {
campaignId?: string;
catalog?: readonly ReportExecution[];
history?: RunnerE2EHistoryIndex;
historySuites?: readonly { id: string; label: string; campaignCount: number }[];
}) {
const h = reportHtml;
const campaign = Boolean(input.catalog);
const label = input.campaignId ?? "Current campaign";
const root = campaign ? reportHistoryHref(input.history, input.campaignId) : "./";
const suites = [...new Map((input.catalog ?? []).map(row => [row.suite.id, row.suite])).values()];
const tree = campaign ? `<nav class="report-suite-tree" aria-label="Campaign suites"><p class="report-side-label">Suites</p>${suites.map(suite => {
const cases = [...new Map(input.catalog!.filter(row => row.suite.id === suite.id).map(row => [row.task.id, row])).values()];
return `<details data-nav-suite="${h(suite.id)}"><summary><a href="#suite-${h(suite.id)}" data-nav-suite-link="${h(suite.id)}">${h(suite.label)}<small>${cases.length}</small></a></summary><ul>${cases.map(row => `<li><a href="?view=case&amp;case=${encodeURIComponent(`${suite.id}.${row.task.id}`)}#execution-${h(row.id)}" data-nav-case="${h(`${suite.id}.${row.task.id}`)}" data-report-case-link="${h(`${suite.id}.${row.task.id}`)}"><span class="case-status-dot" aria-hidden="true"></span><span>${h(row.task.label)}</span></a></li>`).join("")}</ul></details>`;
}).join("")}</nav>` : input.historySuites ? `<nav class="report-suite-catalog" aria-label="E2E suites"><p class="report-side-label">Suites</p><input type="search" data-index-suite-query aria-label="Find suites" placeholder="Find suites…">${input.historySuites.map(suite => `<a href="?view=history&amp;suite=${encodeURIComponent(suite.id)}" data-index-suite-link="${h(suite.id)}"><span>${h(suite.label)}</span><small title="${suite.campaignCount} campaigns">${suite.campaignCount}</small></a>`).join("")}</nav>` : "";
const presets = [...new Set(input.history?.campaigns.map(row => row.preset).filter((preset): preset is string => Boolean(preset)) ?? [])];
return `<div class="report-shell" data-report-scope="${campaign ? "campaign" : "global"}" data-campaign-label="${h(label)}">
<button class="report-drawer-backdrop" type="button" data-nav-close aria-label="Close navigation" hidden></button>
<aside class="report-sidebar" id="report-navigation" aria-label="Report navigation">
<a class="brand-lockup report-brand" href="https://pages.paperclip.ing/evals/" aria-label="Paperclip evaluations">${paperclipReportLogo}<small>Evaluations</small></a>
<div class="report-sidebar-scroll"><nav class="report-global-nav" aria-label="Evaluations">
<a href="${campaign ? "?view=all" : "?view=overview"}" data-nav-view="${campaign ? "all" : "overview"}" aria-current="page"><span aria-hidden="true">◫</span>${campaign ? "All results" : "Overview"}</a>
<a href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/chat-catalog-overview-20261029-175400z/" data-nav-catalog><span aria-hidden="true">▦</span>Chat catalog · Oct 29</a>
<a href="?view=attention" data-nav-view="attention"><span aria-hidden="true">○</span>Needs attention</a>
<a href="${h(root)}?view=history" ${campaign ? "" : 'data-nav-view="history"'}><span aria-hidden="true">↶</span>Campaign history</a>
</nav>${!campaign && presets.length ? `<nav class="report-global-nav" aria-label="Run presets"><p class="report-side-label">Run presets</p>${presets.map(preset => `<a href="?view=history&amp;preset=${encodeURIComponent(preset)}" data-index-preset-link="${h(preset)}">${h(preset === "qa-smoke" ? "QA smoke" : preset)}</a>`).join("")}</nav>` : ""}${tree}</div><div class="report-sidebar-footer">E2E tests${campaign ? `<span>${input.catalog!.length} executions</span>` : ""}</div>
</aside><div class="report-surface">
<header class="report-topbar"><button type="button" class="report-menu-toggle" data-nav-toggle aria-label="Open navigation" aria-controls="report-navigation" aria-expanded="false">☰</button>
<nav class="report-breadcrumbs" aria-label="Breadcrumb" data-nav-breadcrumbs><a href="${h(root)}?view=history">Campaigns</a>${campaign ? `<span aria-hidden="true">/</span><span>${h(label)}</span><span aria-hidden="true">/</span><span aria-current="page">All results</span>` : '<span aria-hidden="true">/</span><span aria-current="page">Overview</span>'}</nav>
${campaign && input.history?.campaigns.length ? `<select class="report-campaign-select" data-nav-campaign aria-label="Campaign">${!input.history.campaigns.some(item => item.campaignId === input.campaignId) ? `<option selected value="">${h(label)}</option>` : ""}${input.history.campaigns.map(item => `<option value="${h(item.publicUrl)}" ${item.campaignId === input.campaignId ? "selected" : ""}>${h(item.campaignId)}</option>`).join("")}</select>` : ""}
</header>`;
}
export const reportShellScript = `
(() => {
const shell = document.querySelector('.report-shell');
const toggle = document.querySelector('[data-nav-toggle]');
const backdrop = document.querySelector('[data-nav-close]');
const close = () => { shell.classList.remove('navigation-open'); toggle.setAttribute('aria-expanded', 'false'); backdrop.hidden = true; };
toggle.addEventListener('click', () => { const open = !shell.classList.contains('navigation-open'); shell.classList.toggle('navigation-open', open); toggle.setAttribute('aria-expanded', String(open)); backdrop.hidden = !open; });
backdrop.addEventListener('click', close);
shell.querySelectorAll('.report-sidebar a').forEach(link => link.addEventListener('click', close));
document.addEventListener('keydown', event => { if(event.key === 'Escape' && shell.classList.contains('navigation-open')) { close(); toggle.focus(); } });
document.querySelector('[data-nav-campaign]')?.addEventListener('change', event => { if(event.target.value) location.href = event.target.value; });
})();
`;
export const reportNavigationStyles = `
:root { --report-sidebar-width: 250px; --navigation-sticky-offset: 64px; }
:root { --report-filter-active:#2563eb; --report-filter-tint:#eff6ff; }
@media(prefers-color-scheme:dark) { :root { --report-filter-active:#60a5fa; --report-filter-tint:#172235; } }
.report-shell [data-filter-active=true] { border-color:var(--report-filter-active); background-color:var(--report-filter-tint); box-shadow:0 0 0 1px var(--report-filter-active); }
.report-shell label:has([data-filter-active=true]) > span { color:var(--report-filter-active); }
html { scroll-behavior: auto; }
[hidden] { display: none !important; }
.report-shell { min-height: 100dvh; }
.report-surface { margin-left: var(--report-sidebar-width); min-width: 0; }
.report-surface main { width: 100%; max-width: 1320px; margin: 0 auto; padding: 32px 36px 72px; }
.report-sidebar { position: fixed; inset: 0 auto 0 0; width: var(--report-sidebar-width); display: flex; flex-direction: column; background: var(--card, var(--raised)); border-right: 1px solid var(--border); z-index: 20; }
.report-brand { min-height: 68px; padding: 20px 16px; display: flex; align-items: center; flex-wrap: wrap; gap: 6px 9px; font-size: 14px; text-decoration: none; }
.report-brand small { font-size: 10px; font-weight: 400; margin-left: auto; color: var(--muted-foreground); }
.report-brand svg { width: auto; height: 24px; flex: none; }
.report-sidebar-scroll { min-height: 0; flex: 1; overflow: auto; padding: 0 10px 24px; scrollbar-width: thin; }
.report-global-nav { display: grid; gap: 3px; }
.report-global-nav a { display: flex; gap: 11px; align-items: center; padding: 9px 10px; font-size: 12px; color: var(--muted-foreground); border-radius: 5px; text-decoration: none; }
.report-global-nav a:hover,.report-suite-tree a:hover { background: var(--raised); color: var(--foreground); }
.report-global-nav a[aria-current=page] { color: var(--foreground); background: var(--raised); }
.report-side-label { padding: 18px 10px 7px; margin: 0; font: 10px var(--font-mono); color: var(--muted-foreground); text-transform: uppercase; letter-spacing: .08em; }
.report-suite-catalog { display: grid; gap: 2px; }
.report-suite-catalog input { min-width: 0; width: calc(100% - 12px); margin: 0 6px 7px; padding: 8px; border: 1px solid var(--border); border-radius: 5px; color: var(--foreground); background: var(--background); font: 12px var(--font-sans); }
.report-suite-catalog a { display: flex; align-items: baseline; justify-content: space-between; gap: 10px; padding: 8px 10px; color: var(--muted-foreground); text-decoration: none; border-radius: 5px; font-size: 12px; }
.report-suite-catalog a span { min-width: 0; overflow-wrap: anywhere; }
.report-suite-catalog a small { flex: none; font: 10px var(--font-mono); }
.report-suite-catalog a:hover,.report-suite-catalog a[aria-current=page] { color: var(--foreground); background: var(--raised); }
.report-suite-tree details > summary { display: flex; align-items: center; gap: 4px; list-style: none; cursor: pointer; padding-left: 4px; }
.report-suite-tree summary::-webkit-details-marker { display: none; }
.report-suite-tree summary::before { content: '›'; width: 12px; flex: none; color: var(--muted-foreground); }
.report-suite-tree details[open] > summary::before { content: '⌄'; }
.report-suite-tree a { display: flex; align-items: center; gap: 8px; min-width: 0; flex: 1; padding: 8px 5px; border-radius: 5px; text-decoration: none; font-size: 12px; color: var(--muted-foreground); }
.report-suite-tree a small { margin-left: auto; font: 10px var(--font-mono); }
.report-suite-tree a[aria-current=page],.report-suite-tree a.in-view { color: var(--foreground); background: var(--raised); }
.report-suite-tree ul { display: grid; gap: 2px; margin: 0; padding: 0 0 8px 22px; list-style: none; }
.report-suite-tree ul a { font-size: 11px; align-items: baseline; }
.report-suite-tree .case-status-dot { flex: 0 0 5px; width: 5px; height: 5px; }
.report-sidebar-footer { display: flex; justify-content: space-between; gap: 10px; border-top: 1px solid var(--border); color: var(--muted-foreground); font-size: 10px; padding: 17px 20px; }
header.report-topbar { position: sticky; top: 0; z-index: 12; display: flex; align-items: center; justify-content: space-between; flex-wrap: nowrap; min-height: 64px; gap: 20px; margin: 0; padding: 12px 36px; border-bottom: 1px solid var(--border); background: var(--background); }
.report-breadcrumbs { display: flex; align-items: center; flex-wrap: wrap; gap: 5px 8px; font-size: 11px; min-width: 0; color: var(--muted-foreground); }
.report-breadcrumbs a { text-decoration: none; }
.report-breadcrumbs a:hover { text-decoration: underline; }
.report-breadcrumbs [aria-current=page] { color: var(--foreground); }
.report-breadcrumbs a,.report-breadcrumbs span { overflow-wrap: anywhere; }
.report-campaign-select { font: 11px var(--font-sans); padding: 8px; color: var(--foreground); background: var(--card); border: 1px solid var(--border); border-radius: 5px; width: 220px; max-width: 30%; flex: none; }
.report-shell .report-surface select {
appearance: none;
-webkit-appearance: none;
padding-inline-end: 34px;
background-image: linear-gradient(45deg, transparent 50%, var(--muted-foreground) 50%), linear-gradient(135deg, var(--muted-foreground) 50%, transparent 50%);
background-position: calc(100% - 16px) center, calc(100% - 12px) center;
background-size: 4px 4px;
background-repeat: no-repeat;
}
.report-menu-toggle { display: none; border: 0; padding: 0; background: transparent; color: var(--foreground); font-size: 20px; cursor: pointer; }
.report-drawer-backdrop { display: none; }
.report-header h1 { font-size: 32px; }
.report-header .lede { font-size: 12px; }
.report-source { display: flex; flex-wrap: wrap; gap: 12px; color: var(--muted-foreground); font-size: 11px; }
.report-header { align-items: start; gap: 24px; }
.report-actions { flex-wrap: wrap; }
.campaign-billing-details { margin: 8px 0 24px; }
.campaign-billing-details > summary { cursor: pointer; color: var(--muted-foreground); font-size: 12px; }
[data-campaign-overview] .campaign-suites { grid-template-columns: repeat(2,minmax(0,1fr)); gap: 0 28px; }
[data-campaign-overview] .campaign-suite-row { grid-template-columns: minmax(0,1fr) auto; gap: 4px 16px; }
[data-campaign-overview] .campaign-suite-state { grid-column: 1/-1; text-align: left; font-size: 10px; }
.report-surface .report-header { grid-template-columns: 1fr; }
.report-surface .results-heading { scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); }
.report-surface .suite-section { scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); margin-top: 56px; }
.report-surface .section-heading h2 { font-size: 26px; }
.report-surface .suite-case-list { grid-template-columns: repeat(2, minmax(0, 1fr)); }
.report-journey { margin: 32px 0 44px; scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); }
.report-journey-heading { display: flex; justify-content: space-between; align-items: baseline; gap: 20px; margin: 0 0 14px; }
.report-journey-heading h3 { margin: 0; font-size: 15px; font-weight: 600; }
.report-journey-heading a { text-decoration: none; }
.report-journey-heading a:hover { text-decoration: underline; }
.report-journey-heading > a { font-size: 11px; color: var(--muted-foreground); white-space: nowrap; }
.execution-previews { display: grid; grid-template-columns: repeat(auto-fit,minmax(min(100%,390px),1fr)); gap: 20px; align-items: start; }
.execution-preview { min-width: 0; }
.execution-preview-label { display: flex; align-items: baseline; flex-wrap: wrap; gap: 8px; margin-bottom: 8px; font-size: 10px; color: var(--muted-foreground); }
.execution-preview-label strong { color: var(--foreground); font-weight: 500; }
.report-surface .profile-sticky { position: static; }
.execution-preview .case { scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); }
.case-open { text-decoration: none; }
.case-open:hover { text-decoration: underline; }
.case-config-options { display: flex; flex-wrap: wrap; gap: 7px; margin-bottom: 16px; }
.case-config-options button { display: flex; align-items: center; gap: 7px; cursor: pointer; padding: 7px 9px; font-size: 11px; color: var(--muted-foreground); background: var(--background); border: 1px solid var(--border); border-radius: 5px; }
.case-config-options button[aria-pressed=true] { color: var(--foreground); background: var(--raised); border-color: var(--border-strong); }
.case-config-options button:focus-visible,.report-menu-toggle:focus-visible { outline: 2px solid currentColor; outline-offset: 3px; }
.case-page-heading { margin-bottom: 26px; }
.case-page-heading h1 { font-size: 28px; margin-bottom: 12px; }
.case-page-heading a { font-size: 12px; color: var(--muted-foreground); }
.report-shell[data-view=case] .report-journey-heading,.report-shell[data-view=case] .suite-case-toc,.report-shell[data-view=case] .suite-section > .section-heading,.report-shell[data-view=case] .suite-summary,.report-shell[data-view=case] .execution-preview-label { display: none; }
.report-shell[data-view=case] .suite-section,.report-shell[data-view=case] .report-journey { margin: 0; }
.report-shell[data-view=case] .execution-previews { display: block; }
.report-shell[data-view=case] .case { scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); }
@media(max-width:1100px) { :root { --report-sidebar-width: 220px; }.report-surface main { padding: 28px 24px 60px; }header.report-topbar { padding: 12px 24px; }.report-campaign-select { max-width: 35%; }.report-surface .suite-summary { grid-template-columns: repeat(3,minmax(0,1fr)); } }
@media(max-width:760px) {
.report-surface { margin-left: 0; }.report-surface main { padding: 24px 17px 56px; }
.report-sidebar { display: none; width: min(290px,85vw); }.navigation-open .report-sidebar { display: flex; }
.navigation-open .report-drawer-backdrop { display: block; position: fixed; inset: 0; border: 0; background: var(--overlay,#0009); z-index: 19; }
header.report-topbar { display: grid; grid-template-columns: 24px minmax(0,1fr); gap: 10px; padding: 12px 17px; }
.report-menu-toggle { display: block; grid-row: 1; grid-column: 1; }.report-campaign-select { grid-row: 1; grid-column: 2; justify-self: end; max-width: 100%; width: 220px; }
.report-breadcrumbs { grid-column: 1/-1; font-size: 10px; }
.report-surface .suite-case-list { grid-template-columns: 1fr; }
[data-campaign-overview] .campaign-suites { grid-template-columns: 1fr; }
.report-surface .suite-summary { grid-template-columns: repeat(2,minmax(0,1fr)); }
.report-surface .report-header h1 { font-size: 28px; }.report-surface .section-heading h2 { font-size: 24px; }
.execution-previews { grid-template-columns: 1fr; }.report-journey-heading { align-items: start; }.report-journey-heading > a { max-width: 80px; white-space: normal; text-align: right; }
.report-surface .report-filters input,.report-surface .report-filters select { font-size: 16px; }
.report-shell[data-view=case] .gallery { grid-template-columns: 1fr; }
}
@media print { .report-sidebar,.report-topbar { display: none; }.report-surface { margin-left: 0; }.report-surface main { padding: 0; } }
`;