mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 08:22:20 +02:00
Add campaign sidebar and case navigation to runner E2E reports (#17010)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E campaigns test runner behavior across agent profiles and environments. > - The retained reports contain many suites, cases, and execution previews. > - A long report makes it hard to find a case or return to the campaign context. > - This pull request adds campaign navigation, case breadcrumbs, and separate global history pages. > - Users can scan all execution previews or inspect one case without losing their place. ## Linked Issues or Issue Description **What existing behavior does this improve?** Navigation in the retained Runner E2E campaign reports and history homepage. **Subsystem affected** Test report infrastructure in `tests/runner-e2e/`. **Current behavior** Reports show a long configuration matrix. The suite index and campaign history share the same page. Case links do not provide a focused inspection view. **Proposed behavior** Global overview, history, and attention pages show global navigation. A specific campaign adds a folding suite tree. All results keeps the full scrollable report. A case opens with breadcrumbs and an execution picker. **Reason and benefit** Users can find a case, compare its executions, and return to their reading position. Global pages do not show suites from an unrelated campaign. **Breaking changes** The report HTML layout changes. Retained result schemas, execution identities, and suite and execution anchors remain unchanged. **Additional context** Refs: #16996. This extends the merged report navigation and local preview work. The duplicate PR and issue search found no open PR for this change. ## What Changed - Add shared report navigation with the Paperclip brand mark and wordmark. - Scope the suite tree to the current campaign. Keep global overview, history, and attention pages separate. - Group all execution previews by suite and case. Reuse the existing evidence cards and image gallery. - Add focused case inspection, configuration selection, breadcrumbs, filter handling, and scroll restoration. - Move historical trends to the global history page. Keep billing details, missing results, and failure evidence available. - Update report documentation and browser regressions for filters, status dots, configuration reloads, breadcrumbs, and scroll restoration. ## Verification - [Final CI](https://github.com/paperclipai/paperclip/actions/runs/39500401411) passes on `89a8b5c290de3cfb9bdc061be3c88a29fd160a62`. Typecheck, build, runner checks, server suites, and all browser shards pass. - All four report browser regressions pass. They cover filters, gallery navigation, repeated suite jumps, the folding sidebar, breadcrumbs, scroll restoration, configuration selection and reload, mobile navigation, and aggregate case status. - The 63 report, history, preview-boundary, and navigation tests pass. Runner E2E typecheck passes with `node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json`. - Manual browser checks covered desktop and a 390-pixel viewport. Retained campaigns covered 209 executions in eight suites, 179 executions in seven suites, and 15 executions in one suite. Browser fixture checks also confirmed filtered inspection and aggregate status. No paid evaluations ran. - The initial report browser tests still targeted the old layout and routes. They were updated for the new navigation. Review also exposed filtered-inspection and aggregate-status defects; both were fixed before the final passing browser run. - The unchanged Cases UI grouping test [failed once on the final head](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118898559197). Its full file also failed locally, while the isolated test passed. The same UI code passed in earlier CI. A [single CI retry](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118903333430) passed all 3,978 UI tests and 286 CLI tests without code changes. The cause remains unresolved. - Two jobs on an intermediate head were interrupted by runner shutdown signals. Those remain interrupted results. The complete final-head CI run supersedes them for merge verification. - Workspace `pnpm -r typecheck` and `pnpm build` also passed locally before the rebase. The full local `pnpm test:run` attempt encountered repeated embedded PostgreSQL startup failures and was stopped. It is not a passing local full-suite result; final CI provides the full verification. ## Risks - Navigation and hash handling can select the wrong case or hide results. Focused tests and browser checks cover these paths. - Historical campaign links must resolve from both the preview and published report locations. - The report layout changes. Graders, prompts, retained outcomes, and billing calculations do not change. ## Model Used - OpenAI Codex with `gpt-6-astra` and `gpt-6.1-sol`. Used reasoning, repository tools, code execution, and browser verification. The runtime did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
2a5a323e57
commit
f89c37791c
12 files changed
+958
-636
No files matched your search
@@ -1,4 +1,4 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
import { expect, test, type Page } from "@playwright/test";
|
||||
|
||||
import { runnerExecutionById, runnerMatrix } from "../runner-e2e/catalog.js";
|
||||
import {
|
||||
@@ -8,6 +8,13 @@ import {
|
||||
import { buildRunnerCampaign } from "../runner-e2e/history.js";
|
||||
import type { MatrixExecution, RunnerE2EResult } from "../runner-e2e/types.js";
|
||||
|
||||
const reportUrl = "http://runner-e2e.test/campaign/";
|
||||
|
||||
async function openReport(page: Page, report: string) {
|
||||
await page.route(`${reportUrl}**`, (route) => route.fulfill({ contentType: "text/html", body: report }));
|
||||
await page.goto(reportUrl);
|
||||
}
|
||||
|
||||
const executionIds = [
|
||||
"core-compatibility.legacy-codex.local.message-marker",
|
||||
"core-compatibility.legacy-codex.daytona.message-marker",
|
||||
@@ -66,14 +73,15 @@ function resultFor(
|
||||
};
|
||||
}
|
||||
|
||||
test("filters matrix results and pages through the filtered screenshot gallery", async ({
|
||||
test("filters execution previews and pages through the filtered screenshot gallery", async ({
|
||||
page,
|
||||
}) => {
|
||||
const catalog = executionIds.map(runnerExecutionById);
|
||||
const entries = catalog.map((execution, index) =>
|
||||
resultFor(execution, index === 1 ? "failed" : "passed"),
|
||||
);
|
||||
await page.setContent(
|
||||
await openReport(
|
||||
page,
|
||||
renderRunnerE2EDashboard({
|
||||
title: "Runner E2E interaction fixture",
|
||||
generatedAt: "2026-09-05T12:01:00.000Z",
|
||||
@@ -92,7 +100,7 @@ test("filters matrix results and pages through the filtered screenshot gallery",
|
||||
await expect(page.locator('select[data-report-suite] option')).toHaveCount(3);
|
||||
await expect(page.locator('select[data-report-profile] option')).toHaveCount(4);
|
||||
await expect(page.locator("[data-report-result]")).toHaveText(
|
||||
"4 of 4 tests shown",
|
||||
"4 of 4 executions shown",
|
||||
);
|
||||
await expect(page.locator("[data-gallery-item]")).toHaveCount(4);
|
||||
const passedCard = page.locator('[data-execution-id="core-compatibility.legacy-codex.local.message-marker"]');
|
||||
@@ -110,32 +118,23 @@ test("filters matrix results and pages through the filtered screenshot gallery",
|
||||
await page.locator('.feature-coverage').getByRole('button', { name: 'Questions and approvals', exact: true }).click();
|
||||
await expect(page.locator('select[data-report-feature]')).toHaveValue('questions-and-approvals');
|
||||
await expect(page.locator('.feature-coverage')).not.toHaveAttribute('open');
|
||||
await expect(page.locator('[data-report-result]')).toHaveText('1 of 4 tests shown');
|
||||
await expect(page.locator('[data-report-result]')).toHaveText('1 of 4 executions shown');
|
||||
await expect(page.locator('.suite-section:visible')).toHaveCount(1);
|
||||
await expect(page.locator('[data-gallery-open]')).toHaveText('View gallery · 1');
|
||||
await page.setViewportSize({ width: 900, height: 900 });
|
||||
await expect(page.locator('[data-report-profile-row]:visible')).toHaveCount(1);
|
||||
await expect(page.locator('.execution-preview:visible')).toHaveCount(1);
|
||||
await page.setViewportSize({ width: 1280, height: 720 });
|
||||
await page.locator('[data-report-reset]').click();
|
||||
|
||||
const firstTable = page.locator(".matrix").first();
|
||||
const widthsBefore = await firstTable
|
||||
.locator("thead th")
|
||||
.evaluateAll((cells) =>
|
||||
cells.map((cell) => cell.getBoundingClientRect().width),
|
||||
);
|
||||
await firstTable.locator("details summary").first().click();
|
||||
const widthsAfter = await firstTable
|
||||
.locator("thead th")
|
||||
.evaluateAll((cells) =>
|
||||
cells.map((cell) => cell.getBoundingClientRect().width),
|
||||
);
|
||||
expect(widthsAfter).toEqual(widthsBefore);
|
||||
const widthBefore = await passedCard.evaluate((card) => card.getBoundingClientRect().width);
|
||||
await passedCard.locator(".case-diagnostics > summary").click();
|
||||
const widthAfter = await passedCard.evaluate((card) => card.getBoundingClientRect().width);
|
||||
expect(widthAfter).toBe(widthBefore);
|
||||
await expect(passedCard.locator(".billing-strip")).toBeVisible();
|
||||
|
||||
await page.locator("[data-report-query]").fill("Legacy Claude");
|
||||
await expect(page.locator("[data-report-result]")).toHaveText(
|
||||
"1 of 4 tests shown",
|
||||
"1 of 4 executions shown",
|
||||
);
|
||||
await expect(page.locator("[data-gallery-open]")).toHaveText(
|
||||
"View gallery · 1",
|
||||
@@ -147,7 +146,7 @@ test("filters matrix results and pages through the filtered screenshot gallery",
|
||||
.locator("select[data-report-profile]")
|
||||
.selectOption("legacy-codex");
|
||||
await expect(page.locator("[data-report-result]")).toHaveText(
|
||||
"2 of 4 tests shown",
|
||||
"2 of 4 executions shown",
|
||||
);
|
||||
await expect(page.locator("[data-gallery-open]")).toHaveText(
|
||||
"View gallery · 2",
|
||||
@@ -206,13 +205,11 @@ function navigationReport(ids: readonly string[], suppliedEntries?: readonly Run
|
||||
|
||||
test("suite links reveal filtered sections and scroll on repeated clicks", async ({ page }) => {
|
||||
const report = navigationReport(executionIds);
|
||||
const reportUrl = "http://runner-e2e.test/campaign/";
|
||||
await page.route(reportUrl, (route) => route.fulfill({ contentType: "text/html", body: report }));
|
||||
await page.goto(reportUrl);
|
||||
await openReport(page, report);
|
||||
|
||||
const core = page.locator("#suite-core-compatibility");
|
||||
const coreHeading = core.getByRole("heading", { name: "Core Runner Compatibility", exact: true });
|
||||
const coreLink = page.getByRole("link", { name: "Core Runner Compatibility", exact: true });
|
||||
const coreLink = page.locator('[data-nav-suite-link="core-compatibility"]');
|
||||
const backToTop = page.getByRole("link", { name: "Back to top", exact: true });
|
||||
await expect(backToTop).toBeHidden();
|
||||
const feature = page.locator("select[data-report-feature]");
|
||||
@@ -223,8 +220,8 @@ test("suite links reveal filtered sections and scroll on repeated clicks", async
|
||||
await expect(core).toBeVisible();
|
||||
await expect(coreHeading).toBeInViewport();
|
||||
await expect(feature).toHaveValue("");
|
||||
await expect(page).toHaveURL(`${reportUrl}#suite-core-compatibility`);
|
||||
await expect(page.locator("[data-report-result]")).toHaveText("4 of 4 tests shown");
|
||||
await expect(page).toHaveURL(`${reportUrl}?view=all#suite-core-compatibility`);
|
||||
await expect(page.locator("[data-report-result]")).toHaveText("4 of 4 executions shown");
|
||||
}
|
||||
|
||||
await page.locator("select[data-report-profile]").selectOption("legacy-codex");
|
||||
@@ -241,7 +238,7 @@ test("suite links reveal filtered sections and scroll on repeated clicks", async
|
||||
await expect(page.locator("select[data-report-profile]")).toHaveValue("legacy-codex");
|
||||
await coreLink.click();
|
||||
|
||||
await page.getByRole("link", { name: "Local Session Integrity", exact: true }).click();
|
||||
await page.locator('[data-nav-suite-link="local-session-integrity"]').click();
|
||||
await expect(page.locator("#suite-local-session-integrity")).toBeVisible();
|
||||
await feature.selectOption("questions-and-approvals");
|
||||
await page.goBack();
|
||||
@@ -249,50 +246,69 @@ test("suite links reveal filtered sections and scroll on repeated clicks", async
|
||||
await expect(feature).toHaveValue("");
|
||||
});
|
||||
|
||||
test("folds the suite index and links each case to a matching configuration", async ({ page }) => {
|
||||
test("folds the suite tree, inspects configurations, and returns to the scrolling report", async ({ page }) => {
|
||||
const ids = [...executionIds, "core-compatibility.legacy-codex.local.ask-question"];
|
||||
const reportUrl = "http://runner-e2e.test/campaign/";
|
||||
await page.route(reportUrl, (route) => route.fulfill({ contentType: "text/html", body: navigationReport(ids) }));
|
||||
await page.goto(reportUrl);
|
||||
const coreIndex = page.locator(".campaign-suite-details").filter({
|
||||
has: page.getByRole("link", { name: "Core Runner Compatibility", exact: true }),
|
||||
});
|
||||
await expect(coreIndex).not.toHaveAttribute("open");
|
||||
await coreIndex.getByRole("link", { name: "Core Runner Compatibility", exact: true }).click();
|
||||
await openReport(page, navigationReport(ids));
|
||||
const coreIndex = page.locator('[data-nav-suite="core-compatibility"]');
|
||||
const coreLink = coreIndex.locator('[data-nav-suite-link]');
|
||||
await expect(coreIndex).not.toHaveAttribute("open");
|
||||
await coreLink.click();
|
||||
await expect(coreIndex).toHaveAttribute("open");
|
||||
await coreIndex.locator("summary").focus();
|
||||
await page.keyboard.press("Enter");
|
||||
await expect(coreIndex).not.toHaveAttribute("open");
|
||||
await page.keyboard.press("Enter");
|
||||
await expect(coreIndex).toHaveAttribute("open");
|
||||
await expect(coreIndex.locator(".suite-case-list a")).toHaveCount(2);
|
||||
await expect(coreIndex.locator("[data-nav-case]")).toHaveCount(2);
|
||||
const sectionToc = page.locator("#suite-core-compatibility .suite-case-toc");
|
||||
await expect(sectionToc.locator(".suite-case-list a")).toHaveCount(2);
|
||||
|
||||
const basicLink = coreIndex.getByRole("link", { name: "Basic response", exact: true });
|
||||
const profile = page.locator("select[data-report-profile]");
|
||||
const feature = page.locator("select[data-report-feature]");
|
||||
const crumbs = page.getByRole("navigation", { name: "Breadcrumb", exact: true });
|
||||
const returnToCampaign = crumbs.getByRole("link", { name: "navigation-fixture", exact: true });
|
||||
await profile.selectOption("legacy-claude");
|
||||
const claudeResult = page.locator('[data-execution-id="core-compatibility.legacy-claude.local.message-marker"]');
|
||||
const readingPosition = await page.evaluate(() => window.scrollY);
|
||||
await basicLink.click();
|
||||
await expect(claudeResult.locator(".case-heading")).toBeInViewport();
|
||||
const claudeResult = page.locator('[data-execution-id="core-compatibility.legacy-claude.local.message-marker"]');
|
||||
await expect(claudeResult).toBeVisible();
|
||||
await expect(page.locator("[data-report-case]:visible")).toHaveCount(1);
|
||||
await expect(page.getByRole("button", { name: "Legacy Claude · Isolated local", exact: true })).toHaveAttribute("aria-pressed", "true");
|
||||
await expect(page.locator(".report-filters")).toBeHidden();
|
||||
expect(new URL(page.url()).searchParams.get("view")).toBe("case");
|
||||
expect(new URL(page.url()).hash).toBe("#execution-core-compatibility.legacy-claude.local.message-marker");
|
||||
await returnToCampaign.click();
|
||||
await expect(profile).toHaveValue("legacy-claude");
|
||||
await expect(page).toHaveURL(`${reportUrl}#execution-core-compatibility.legacy-claude.local.message-marker`);
|
||||
await expect.poll(() => page.evaluate(() => window.scrollY)).toBe(readingPosition);
|
||||
|
||||
for (let click = 0; click < 2; click += 1) {
|
||||
await feature.selectOption("questions-and-approvals");
|
||||
await basicLink.click();
|
||||
await expect(feature).toHaveValue("");
|
||||
await expect(profile).toHaveValue("");
|
||||
await expect(page.locator('[data-execution-id="core-compatibility.legacy-codex.local.message-marker"] .case-heading')).toBeInViewport();
|
||||
}
|
||||
await feature.selectOption("questions-and-approvals");
|
||||
await basicLink.click();
|
||||
await expect(page.locator('[data-execution-id="core-compatibility.legacy-codex.local.message-marker"]')).toBeVisible();
|
||||
await expect(page.locator("[data-nav-configuration]:visible")).toHaveCount(3);
|
||||
await crumbs.getByRole("link", { name: "Core Runner Compatibility", exact: true }).click();
|
||||
await expect(feature).toHaveValue("");
|
||||
await expect(profile).toHaveValue("");
|
||||
await expect(page.locator("[data-report-case]:visible")).toHaveCount(5);
|
||||
await sectionToc.getByRole("link", { name: "Ask mode question", exact: true }).click();
|
||||
await expect(page.locator('[data-execution-id="core-compatibility.legacy-codex.local.ask-question"] .case-heading')).toBeInViewport();
|
||||
await sectionToc.getByRole("link", { name: "Back to index", exact: false }).click();
|
||||
await expect(coreIndex.locator("summary")).toBeInViewport();
|
||||
await sectionToc.getByRole("link", { name: "Back to overview", exact: false }).click();
|
||||
await expect(page.locator("#overview")).toBeInViewport();
|
||||
|
||||
await page.setViewportSize({ width: 390, height: 844 });
|
||||
await expect(coreIndex).toBeHidden();
|
||||
await page.getByRole("button", { name: "Open navigation", exact: true }).click();
|
||||
await expect(basicLink).toBeVisible();
|
||||
expect(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth)).toBe(true);
|
||||
await basicLink.click();
|
||||
await expect(coreIndex).toBeHidden();
|
||||
const daytona = page.getByRole("button", { name: "Legacy Codex · Daytona sandbox", exact: true });
|
||||
await daytona.click();
|
||||
await page.reload();
|
||||
await expect(daytona).toHaveAttribute("aria-pressed", "true");
|
||||
await expect(page.locator('[data-execution-id="core-compatibility.legacy-codex.daytona.message-marker"]')).toBeVisible();
|
||||
await expect(page.locator("[data-report-case]:visible")).toHaveCount(1);
|
||||
await crumbs.getByRole("link", { name: "Core Runner Compatibility", exact: true }).click();
|
||||
const backToTop = page.getByRole("link", { name: "Back to top", exact: true });
|
||||
await expect(backToTop).toBeVisible();
|
||||
expect(await backToTop.evaluate((element) => {
|
||||
@@ -306,28 +322,31 @@ test("folds the suite index and links each case to a matching configuration", as
|
||||
test("case status dots summarize matching results and open failures", async ({ page }) => {
|
||||
const ids = [...executionIds, "core-compatibility.legacy-codex.local.ask-question"];
|
||||
const entries = executionIds.map((id, index) => resultFor(runnerExecutionById(id), index === 1 ? "failed" : "passed"));
|
||||
const reportUrl = "http://runner-e2e.test/campaign/";
|
||||
await page.route(reportUrl, (route) => route.fulfill({ contentType: "text/html", body: navigationReport(ids, entries) }));
|
||||
await page.goto(reportUrl);
|
||||
await openReport(page, navigationReport(ids, entries));
|
||||
const toc = page.locator("#suite-core-compatibility .suite-case-toc");
|
||||
const basic = toc.getByRole("link", { name: "Basic response", exact: true });
|
||||
const missing = toc.getByRole("link", { name: "Ask mode question", exact: true });
|
||||
const topLink = page.locator('.campaign-case-index [data-report-case-link="core-compatibility.message-marker"]');
|
||||
const topLink = page.locator('.report-suite-tree [data-report-case-link="core-compatibility.message-marker"]');
|
||||
await expect(basic).toHaveAttribute("data-case-status", "failed");
|
||||
await expect(topLink).toHaveAttribute("data-case-status", "failed");
|
||||
await expect(basic).toHaveAccessibleDescription("Failed · 2/3 passed · 1 failed — All configurations");
|
||||
await expect(basic).toHaveAccessibleDescription("failed · 2/3 configurations passed");
|
||||
await expect(missing).toHaveAttribute("data-case-status", "missing");
|
||||
await expect(missing).toHaveAttribute("title", "Missing result · 0/1 passed · 1 missing — All configurations");
|
||||
await expect(missing).toHaveAttribute("title", "missing · 0/1 configurations passed");
|
||||
const red = await basic.locator(".case-status-dot").evaluate((element) => getComputedStyle(element).backgroundColor);
|
||||
|
||||
const profile = page.locator("select[data-report-profile]");
|
||||
await profile.selectOption("legacy-claude");
|
||||
await expect(basic).toHaveAttribute("data-case-status", "passed");
|
||||
await expect(topLink).toHaveAttribute("data-case-status", "passed");
|
||||
await expect(basic).toHaveAccessibleDescription("Passed · 1/1 passed — Matching configurations");
|
||||
await expect(basic).toHaveAccessibleDescription("passed · 1/1 configurations passed");
|
||||
const green = await basic.locator(".case-status-dot").evaluate((element) => getComputedStyle(element).backgroundColor);
|
||||
expect(green).not.toBe(red);
|
||||
await expect(missing).toHaveAttribute("title", /none match current filters/);
|
||||
await expect(missing).toHaveAttribute("title", /no matching configurations/);
|
||||
await page.locator('[data-report-journey="core-compatibility.message-marker"]').getByRole("link", { name: "Inspect case ↗", exact: true }).click();
|
||||
await expect(page.locator('[data-execution-id="core-compatibility.legacy-claude.local.message-marker"]')).toBeVisible();
|
||||
await expect(page.locator("[data-report-case]:visible")).toHaveCount(1);
|
||||
await expect(topLink).toHaveAttribute("data-case-status", "passed");
|
||||
await page.getByRole("navigation", { name: "Breadcrumb", exact: true }).getByRole("link", { name: "navigation-fixture", exact: true }).click();
|
||||
|
||||
await profile.selectOption("legacy-codex");
|
||||
await expect(basic).toHaveAttribute("data-case-status", "failed");
|
||||
@@ -335,6 +354,12 @@ test("case status dots summarize matching results and open failures", async ({ p
|
||||
await expect(basic).toHaveAttribute("data-case-status", "passed");
|
||||
await page.locator("[data-report-reset]").click();
|
||||
await basic.click();
|
||||
await expect(page).toHaveURL(`${reportUrl}#execution-core-compatibility.legacy-codex.daytona.message-marker`);
|
||||
await expect(page).toHaveURL(`${reportUrl}?view=all#execution-core-compatibility.legacy-codex.daytona.message-marker`);
|
||||
await expect(page.locator('[data-execution-id="core-compatibility.legacy-codex.daytona.message-marker"] .case-heading')).toBeInViewport();
|
||||
await page.locator('[data-execution-id="core-compatibility.legacy-codex.local.message-marker"] .case-open').click();
|
||||
await expect(page.locator("[data-report-case]:visible")).toHaveCount(1);
|
||||
await expect(topLink).toHaveAttribute("data-case-status", "failed");
|
||||
await expect(topLink).toHaveAccessibleDescription("failed · 2/3 configurations passed");
|
||||
await page.getByRole("button", { name: "Legacy Codex · Daytona sandbox", exact: true }).click();
|
||||
await expect(topLink).toHaveAttribute("data-case-status", "failed");
|
||||
});
|
||||
@@ -778,15 +778,19 @@ captures remain failed evidence; qualifying a fix requires a new live attempt.
|
||||
|
||||
Every completed local campaign also writes
|
||||
`tests/runner-e2e/results/<campaign>/dashboard.html`. The self-contained page
|
||||
shows the complete profile/environment grid with screenshot thumbnails.
|
||||
shows every selected execution with screenshot thumbnails, grouped by suite and case.
|
||||
The history homepage and campaign report render their campaign summaries as
|
||||
selectable HTML text, with suite links, pass counts, status, and timing.
|
||||
Campaign reports render only selected executions, including selected cases with
|
||||
missing evidence or startup failures. Search and feature/suite/profile filters
|
||||
replace the full-catalog sticky index.
|
||||
Each suite in the campaign report summary folds open to a list of its selected cases.
|
||||
The same case links appear at the start of the suite, with a return to the top
|
||||
index. Cases appear once across agent/environment configurations; links prefer
|
||||
sit alongside a folding suite/case sidebar scoped to the current campaign.
|
||||
Global Overview, Campaign history, and Needs attention pages have no suite tree.
|
||||
All results is one continuous report with an overview and all execution previews.
|
||||
Suite links jump into the report; sidebar case links open a focused case view
|
||||
with a configuration picker. Clickable Campaign / Suite / Case breadcrumbs
|
||||
return to the report and preserve its reading position. Suite contents also link
|
||||
to execution previews, with a return to the overview and a bottom-right Back to top.
|
||||
Cases appear once in the index across agent/environment configurations; links prefer
|
||||
a result matching the active filters and clear conflicting filters when needed
|
||||
to reveal their destination.
|
||||
Case links show a green status dot when all matching configurations passed,
|
||||
|
||||
+38
-522
@@ -1,3 +1,6 @@
|
||||
import { html, durationLabel, tokenLabel, llmCostLabel, usdLabel } from "./report-format.js";
|
||||
import { renderReportShell, reportNavigationStyles, reportShellScript } from "./report-navigation.js";
|
||||
import { reportBrowserScript } from "./report-browser.js";
|
||||
import { isIncompleteFirstTaskResult, renderCaseOutcome } from "./case-outcome.js";
|
||||
import { campaignSuiteSummaryStyles, renderCampaignSuiteSummary, renderSuiteCaseIndex } from "./campaign-summary.js";
|
||||
import { journeyCapabilities, journeyFeatureIds } from "./journey-features.js";
|
||||
@@ -9,7 +12,6 @@ import {
|
||||
type ReportExecution,
|
||||
} from "./report-catalog.js";
|
||||
import type {
|
||||
RunnerE2EBillingSummary,
|
||||
RunnerE2ECampaign,
|
||||
RunnerE2EHistoryIndex,
|
||||
RunnerE2EResult,
|
||||
@@ -17,7 +19,6 @@ import type {
|
||||
} from "./types.js";
|
||||
import {
|
||||
aggregateCampaignBilling,
|
||||
billingCoverageLabel,
|
||||
summarizeExecutionBilling,
|
||||
} from "./billing.js";
|
||||
|
||||
@@ -46,36 +47,6 @@ interface ResolvedScreenshot {
|
||||
href: string;
|
||||
}
|
||||
|
||||
function html(value: unknown) {
|
||||
return String(value ?? "")
|
||||
.replaceAll("&", "&")
|
||||
.replaceAll("<", "<")
|
||||
.replaceAll(">", ">")
|
||||
.replaceAll('"', """)
|
||||
.replaceAll("'", "'");
|
||||
}
|
||||
|
||||
function durationLabel(durationMs: number) {
|
||||
if (durationMs < 1_000) return `${durationMs}ms`;
|
||||
const seconds = Math.round(durationMs / 1_000);
|
||||
if (seconds < 60) return `${seconds}s`;
|
||||
return `${Math.floor(seconds / 60)}m ${seconds % 60}s`;
|
||||
}
|
||||
|
||||
function tokenLabel(value: number, llm?: RunnerE2EBillingSummary["llm"]) {
|
||||
const formatted = new Intl.NumberFormat("en-US").format(value);
|
||||
return llm ? billingCoverageLabel(formatted, llm.runsWithTokenUsage, llm.runCount) : formatted;
|
||||
}
|
||||
|
||||
function llmCostLabel(value: number, llm: RunnerE2EBillingSummary["llm"]) {
|
||||
return billingCoverageLabel(usdLabel(value), llm.runsWithReportedCost, llm.runCount);
|
||||
}
|
||||
|
||||
function usdLabel(value: number | null) {
|
||||
if (value === null) return "Unknown";
|
||||
return `$${value.toFixed(value < 0.01 ? 6 : 4)}`;
|
||||
}
|
||||
|
||||
function safeEvidenceHref(base: string | undefined, relative: string) {
|
||||
if (!base || /^(?:[a-z]+:|\/\/)/i.test(base)) return null;
|
||||
const cleanBase = base
|
||||
@@ -297,6 +268,7 @@ function renderCase(
|
||||
class="case case-${state}"
|
||||
data-execution-id="${html(execution.id)}"
|
||||
data-report-case
|
||||
data-report-label="${html(execution.task.label)}"
|
||||
data-report-case-key="${html(`${execution.suite.id}.${execution.task.id}`)}"
|
||||
data-report-suite="${html(execution.suite.id)}"
|
||||
data-report-features="${html(journeyFeatureIds(execution).join(" "))}"
|
||||
@@ -306,7 +278,7 @@ function renderCase(
|
||||
data-report-search="${html(searchText)}"
|
||||
>
|
||||
<div class="case-heading">
|
||||
<strong>${html(execution.task.label)}</strong>
|
||||
<strong><a class="case-open" data-nav-case="${html(`${execution.suite.id}.${execution.task.id}`)}" href="?view=case&case=${encodeURIComponent(`${execution.suite.id}.${execution.task.id}`)}#execution-${html(execution.id)}">${html(execution.task.label)}</a></strong>
|
||||
<span class="status">${html(label)}</span>
|
||||
</div>
|
||||
${features.length ? `<div class="case-features">${features.map((feature) =>
|
||||
@@ -346,230 +318,7 @@ function renderCase(
|
||||
</article>`;
|
||||
}
|
||||
|
||||
function renderTrendChart(input: {
|
||||
history: RunnerE2EHistoryIndex;
|
||||
label: string;
|
||||
value(campaign: RunnerE2EHistoryIndex["campaigns"][number]): number | null;
|
||||
format(value: number): string;
|
||||
include?(campaign: RunnerE2EHistoryIndex["campaigns"][number]): boolean;
|
||||
fingerprint?(campaign: RunnerE2EHistoryIndex["campaigns"][number]): string;
|
||||
}) {
|
||||
const campaigns = input.history.campaigns
|
||||
.filter(input.include ?? ((campaign) => campaign.complete))
|
||||
.slice(0, 20)
|
||||
.reverse();
|
||||
if (campaigns.length === 0) {
|
||||
return `<article class="trend-card"><span>${html(input.label)}</span><strong>No complete campaigns</strong></article>`;
|
||||
}
|
||||
const rawValues = campaigns.map(input.value);
|
||||
const values = rawValues.filter((value): value is number => value !== null);
|
||||
if (values.length !== rawValues.length) {
|
||||
const latest = rawValues.at(-1);
|
||||
return `<article class="trend-card"><span>${html(input.label)}</span><strong>${latest == null ? "Unknown" : html(input.format(latest))}</strong><small>Unknown measurements are not plotted. Inspect campaign billing and reserved judge budgets.</small></article>`;
|
||||
}
|
||||
const maximum = Math.max(...values, 1);
|
||||
const pointRows = values.map((value, index) => {
|
||||
const x =
|
||||
campaigns.length === 1 ? 50 : (index / (campaigns.length - 1)) * 100;
|
||||
const y = 96 - (value / maximum) * 88;
|
||||
return {
|
||||
point: `${x.toFixed(2)},${y.toFixed(2)}`,
|
||||
x,
|
||||
y,
|
||||
fingerprint: input.fingerprint?.(campaigns[index]!) ?? "stable",
|
||||
};
|
||||
});
|
||||
const segments = pointRows.reduce<Array<typeof pointRows>>(
|
||||
(groups, point) => {
|
||||
const current = groups.at(-1);
|
||||
if (!current || current.at(-1)?.fingerprint !== point.fingerprint) {
|
||||
groups.push([point]);
|
||||
} else {
|
||||
current.push(point);
|
||||
}
|
||||
return groups;
|
||||
},
|
||||
[],
|
||||
);
|
||||
const definitionCount = new Set(pointRows.map((point) => point.fingerprint))
|
||||
.size;
|
||||
const latest = values.at(-1) ?? 0;
|
||||
return `<article class="trend-card">
|
||||
<span>${html(input.label)}</span>
|
||||
<strong>${html(input.format(latest))}</strong>
|
||||
<svg viewBox="0 0 100 100" preserveAspectRatio="none" role="img" aria-label="${html(input.label)} across ${campaigns.length} complete campaign(s)">
|
||||
${segments.map((segment) => `<polyline points="${segment.map((point) => point.point).join(" ")}" fill="none" vector-effect="non-scaling-stroke" />`).join("")}
|
||||
${pointRows.map((point) => `<circle cx="${point.x.toFixed(2)}" cy="${point.y.toFixed(2)}" r="1.3" vector-effect="non-scaling-stroke" />`).join("")}
|
||||
</svg>
|
||||
<small>${campaigns.length} complete campaign${campaigns.length === 1 ? "" : "s"} · ${definitionCount} definition${definitionCount === 1 ? "" : "s"}</small>
|
||||
</article>`;
|
||||
}
|
||||
|
||||
function suiteSummaryFor(
|
||||
campaign: RunnerE2EHistoryIndex["campaigns"][number],
|
||||
suiteId: string,
|
||||
) {
|
||||
return campaign.suites.find((suite) => suite.suiteId === suiteId);
|
||||
}
|
||||
|
||||
function renderHistory(history: RunnerE2EHistoryIndex | undefined) {
|
||||
if (!history || history.campaigns.length === 0) {
|
||||
return `<section class="history-section" id="history"><div class="section-heading"><div><p class="eyebrow">History</p><h2>Campaign trends</h2></div><p>No historical campaigns have been published yet.</p></div></section>`;
|
||||
}
|
||||
const suiteIds = [
|
||||
...new Set(
|
||||
history.campaigns.flatMap((campaign) =>
|
||||
campaign.suites.map((suite) => suite.suiteId),
|
||||
),
|
||||
),
|
||||
];
|
||||
const charts = [
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Observed + estimated cost",
|
||||
value: (campaign) => campaign.billing.observedAndEstimatedCostUsd,
|
||||
format: usdLabel,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Total tokens",
|
||||
value: (campaign) => campaign.billing.llm.totalTokens,
|
||||
format: tokenLabel,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Agent execution time",
|
||||
value: (campaign) => campaign.billing.agentRunDurationMs,
|
||||
format: durationLabel,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Daytona lease time",
|
||||
value: (campaign) => campaign.billing.leaseDurationMs,
|
||||
format: durationLabel,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Pass rate",
|
||||
value: (campaign) =>
|
||||
campaign.selected > 0 ? (campaign.passed / campaign.selected) * 100 : 0,
|
||||
format: (value) => `${value.toFixed(1)}%`,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
].join("");
|
||||
const suiteCharts = suiteIds
|
||||
.map((suiteId) => {
|
||||
const include = (campaign: RunnerE2EHistoryIndex["campaigns"][number]) =>
|
||||
suiteSummaryFor(campaign, suiteId)?.complete === true;
|
||||
const value = (
|
||||
campaign: RunnerE2EHistoryIndex["campaigns"][number],
|
||||
metric: "cost" | "tokens" | "agent" | "lease" | "passRate",
|
||||
) => {
|
||||
const suite = suiteSummaryFor(campaign, suiteId);
|
||||
if (!suite) return 0;
|
||||
if (metric === "cost") return suite.billing.observedAndEstimatedCostUsd;
|
||||
if (metric === "tokens") return suite.billing.llm.totalTokens;
|
||||
if (metric === "agent") return suite.billing.agentRunDurationMs;
|
||||
if (metric === "lease") return suite.billing.leaseDurationMs;
|
||||
return suite.selected > 0 ? (suite.passed / suite.selected) * 100 : 0;
|
||||
};
|
||||
const fingerprint = (
|
||||
campaign: RunnerE2EHistoryIndex["campaigns"][number],
|
||||
) => suiteSummaryFor(campaign, suiteId)?.suiteDefinitionHash ?? "unknown";
|
||||
return `<section class="suite-trends" data-history-suite-trends="${html(suiteId)}">
|
||||
<div class="suite-trends-heading"><h3>${html(suiteId)}</h3><span>Complete suite selections only; lines break at definition changes.</span></div>
|
||||
<div class="trend-grid">
|
||||
${renderTrendChart({ history, label: "Suite cost", value: (campaign) => value(campaign, "cost"), format: usdLabel, include, fingerprint })}
|
||||
${renderTrendChart({ history, label: "Suite tokens", value: (campaign) => value(campaign, "tokens"), format: tokenLabel, include, fingerprint })}
|
||||
${renderTrendChart({ history, label: "Suite agent time", value: (campaign) => value(campaign, "agent"), format: durationLabel, include, fingerprint })}
|
||||
${renderTrendChart({ history, label: "Suite lease time", value: (campaign) => value(campaign, "lease"), format: durationLabel, include, fingerprint })}
|
||||
${renderTrendChart({ history, label: "Suite pass rate", value: (campaign) => value(campaign, "passRate"), format: (metric) => `${metric.toFixed(1)}%`, include, fingerprint })}
|
||||
</div>
|
||||
</section>`;
|
||||
})
|
||||
.join("");
|
||||
const rows = history.campaigns
|
||||
.map((campaign) => {
|
||||
const status = campaign.failed > 0 ? "failed" : (campaign.incomplete ?? 0) > 0 ? "incomplete" : "passed";
|
||||
const sha = campaign.source.sha;
|
||||
const searchable = [
|
||||
campaign.campaignId,
|
||||
sha,
|
||||
campaign.source.ref,
|
||||
...campaign.executions.flatMap((execution) => [
|
||||
execution.suiteId,
|
||||
execution.profileId,
|
||||
execution.model,
|
||||
execution.environmentId,
|
||||
execution.caseId,
|
||||
execution.status,
|
||||
]),
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(" ")
|
||||
.toLowerCase();
|
||||
return `<tr data-history-campaign data-history-date="${html(campaign.generatedAt.slice(0, 10))}" data-history-status="${status}" data-history-complete="${campaign.complete}" data-history-suites="${html(campaign.suites.map((suite) => suite.suiteId).join(" "))}" data-history-search="${html(searchable)}">
|
||||
<td><a href="${html(campaign.publicUrl)}">${html(campaign.campaignId)}</a><small>${html(new Date(campaign.generatedAt).toLocaleString("en-US", { timeZone: "UTC" }))} UTC</small></td>
|
||||
<td>${sha ? `<code>${html(sha.slice(0, 10))}</code>` : "Unknown"}<small>${html(campaign.source.ref ?? "unknown ref")}</small></td>
|
||||
<td><span class="status history-${status}">${status}</span><small>${campaign.passed}/${campaign.selected} passed${campaign.incomplete ? ` · ${campaign.incomplete} incomplete` : ""} · ${campaign.complete ? "complete" : "partial"}</small></td>
|
||||
<td>${html(tokenLabel(campaign.billing.llm.inputTokens, campaign.billing.llm))} / ${html(tokenLabel(campaign.billing.llm.outputTokens, campaign.billing.llm))}<small>input / output · ${html(tokenLabel(campaign.billing.llm.cachedInputTokens, campaign.billing.llm))} cached</small></td>
|
||||
<td>${html(llmCostLabel(campaign.billing.reportedLlmCostUsd, campaign.billing.llm))}<small>${html(usdLabel(campaign.billing.estimatedRuntimeCostUsd))} runtime estimate</small></td>
|
||||
<td>${html(durationLabel(campaign.billing.agentRunDurationMs))}<small>${html(durationLabel(campaign.billing.leaseDurationMs))} lease</small></td>
|
||||
</tr>`;
|
||||
})
|
||||
.join("");
|
||||
const latest = history.campaigns.find(
|
||||
(campaign) => campaign.campaignId === history.latestCampaignId,
|
||||
);
|
||||
const latestGreen = history.campaigns.find(
|
||||
(campaign) => campaign.campaignId === history.latestGreenCampaignId,
|
||||
);
|
||||
return `<section class="history-section" id="history">
|
||||
<div class="section-heading"><div><p class="eyebrow">History</p><h2>Campaign trends</h2></div><p>Complete campaigns are compared by default. Partial smoke runs remain searchable and are labeled explicitly.</p></div>
|
||||
<nav class="history-pointers" aria-label="Campaign pointers">
|
||||
${latest ? `<a href="${html(latest.publicUrl)}"><span>Latest run</span><strong>${html(latest.campaignId)}</strong></a>` : ""}
|
||||
${latestGreen ? `<a href="${html(latestGreen.publicUrl)}"><span>Latest green</span><strong>${html(latestGreen.campaignId)}</strong></a>` : ""}
|
||||
</nav>
|
||||
<div class="trend-grid">${charts}</div>
|
||||
${suiteCharts}
|
||||
<div class="history-filters">
|
||||
<label>Search <input type="search" data-history-query placeholder="SHA, model, profile, case"></label>
|
||||
<label>Suite <select data-history-suite><option value="">All suites</option>${suiteIds.map((suiteId) => `<option value="${html(suiteId)}">${html(suiteId)}</option>`).join("")}</select></label>
|
||||
<label>Status <select data-history-status><option value="">All statuses</option><option value="passed">Passed</option><option value="failed">Failed</option><option value="incomplete">Incomplete</option></select></label>
|
||||
<label>From <input type="date" data-history-from></label>
|
||||
<label>Through <input type="date" data-history-through></label>
|
||||
<label class="history-checkbox"><input type="checkbox" data-history-partial> Include partial campaigns</label>
|
||||
</div>
|
||||
<div class="history-table-wrap"><table class="history-table"><thead><tr><th>Campaign</th><th>Paperclip SHA</th><th>Result</th><th>Tokens</th><th>Cost</th><th>Execution</th></tr></thead><tbody>${rows}</tbody></table></div>
|
||||
<p class="history-empty" data-history-empty hidden>No campaigns match these filters.</p>
|
||||
</section>`;
|
||||
}
|
||||
|
||||
function renderSuiteMatrix(input: {
|
||||
function renderSuiteResults(input: {
|
||||
suiteCatalog: readonly ReportExecution[];
|
||||
expected: ReadonlySet<string>;
|
||||
entryById: ReadonlyMap<string, RunnerDashboardEntry>;
|
||||
@@ -577,52 +326,6 @@ function renderSuiteMatrix(input: {
|
||||
}) {
|
||||
const suite = input.suiteCatalog[0]?.suite;
|
||||
if (!suite) return "";
|
||||
const profiles = [
|
||||
...new Map(
|
||||
input.suiteCatalog.map((execution) => [
|
||||
execution.profile.id,
|
||||
execution.profile,
|
||||
]),
|
||||
).values(),
|
||||
];
|
||||
const environments = [
|
||||
...new Map(
|
||||
input.suiteCatalog.map((execution) => [
|
||||
execution.environment.id,
|
||||
execution.environment,
|
||||
]),
|
||||
).values(),
|
||||
];
|
||||
const rows = profiles
|
||||
.map((profile, profileIndex) => {
|
||||
const columns = environments
|
||||
.map((environment) => {
|
||||
const executions = input.suiteCatalog.filter(
|
||||
(execution) =>
|
||||
execution.profile.id === profile.id &&
|
||||
execution.environment.id === environment.id,
|
||||
);
|
||||
return `<td>
|
||||
<div class="mobile-environment-header"><strong>${html(environment.label)}</strong><span>${html(environment.provider)} · ${html(environment.expectedExecutionTarget.kind)}</span></div>
|
||||
<div class="case-stack">${executions.map((execution) => renderCase(execution, input.expected, input.entryById)).join("")}</div>
|
||||
</td>`;
|
||||
})
|
||||
.join("");
|
||||
return `<tr data-report-profile-row><th scope="row" class="profile-cell"><div class="profile-sticky">
|
||||
<span class="agent-capsule agent-${(profileIndex % 10) + 1}" aria-hidden="true"></span>
|
||||
<span class="profile-copy"><span><strong>${html(profile.label)}</strong><span class="generation">${html(profile.generation)}</span></span><small>${html(profile.provider)} · ${html(suite.id === "first-task" ? "Production onboarding default" : profile.model)}</small></span>
|
||||
</div></th>${columns}</tr>`;
|
||||
})
|
||||
.join("");
|
||||
const environmentHeaders = environments
|
||||
.map(
|
||||
(environment) =>
|
||||
`<th scope="col"><span class="environment-label">${html(environment.label)}</span><small>${html(environment.provider)} · ${html(environment.expectedExecutionTarget.kind)}</small></th>`,
|
||||
)
|
||||
.join("");
|
||||
const selected = input.suiteCatalog.filter((execution) =>
|
||||
input.expected.has(execution.id),
|
||||
).length;
|
||||
const summary = input.summary;
|
||||
const summaryHtml = summary
|
||||
? `<div class="suite-summary" aria-label="${html(suite.label)} current campaign summary">
|
||||
@@ -633,19 +336,27 @@ function renderSuiteMatrix(input: {
|
||||
<div><span>Execution</span><strong>${summary.executed}/${summary.selected}</strong><small>${summary.retries} retries · cleanup ${summary.cleanupPassed ? "passed" : "failed"}</small></div>
|
||||
</div>`
|
||||
: "";
|
||||
return `<section class="suite-section" id="suite-${html(suite.id)}">
|
||||
const journeys = [...new Map(input.suiteCatalog.map(row => [row.task.id, row.task])).values()];
|
||||
return `<section class="suite-section" id="suite-${html(suite.id)}" data-report-suite-section="${html(suite.id)}" data-suite-label="${html(suite.label)}">
|
||||
<div class="section-heading"><div><p class="eyebrow">Test suite</p><h2>${html(suite.label)}</h2></div><p>${html(suite.description)}</p></div>
|
||||
${summaryHtml}
|
||||
<nav class="suite-case-toc" aria-label="${html(suite.label)} cases">
|
||||
<div class="suite-case-toc-heading"><h3>Cases</h3><a href="#overview">Back to index ↑</a></div>
|
||||
<div class="suite-case-toc-heading"><h3>Cases</h3><a href="#overview">Back to overview ↑</a></div>
|
||||
${renderSuiteCaseIndex(input.suiteCatalog)}
|
||||
</nav>
|
||||
<div class="table-kicker"><strong>Configuration matrix</strong><span>${profiles.length} profiles · ${environments.length} environments · ${selected} selected</span></div>
|
||||
<div class="table-wrap"><table class="matrix"><colgroup><col class="profile-column">${environments.map(() => '<col class="environment-column">').join("")}</colgroup><thead><tr><th scope="col">Agent profile</th>${environmentHeaders}</tr></thead><tbody>${rows}</tbody></table></div>
|
||||
${journeys.map(task => {
|
||||
const key = `${suite.id}.${task.id}`;
|
||||
const executions = input.suiteCatalog.filter(row => row.task.id === task.id);
|
||||
const href = `?view=case&case=${encodeURIComponent(key)}`;
|
||||
return `<section class="report-journey" id="journey-${html(key)}" data-report-journey="${html(key)}">
|
||||
<div class="report-journey-heading"><h3><a href="${html(href)}" data-nav-case="${html(key)}" data-report-case-link="${html(key)}">${html(task.label)}</a></h3><a href="${html(href)}" data-nav-case="${html(key)}" data-report-case-link="${html(key)}">Inspect case ↗</a></div>
|
||||
<div class="case-config-options" aria-label="Execution configurations" hidden>${executions.map(row => `<button type="button" data-nav-configuration="${html(row.id)}" aria-pressed="false"><span class="case-status-dot" aria-hidden="true"></span>${html(row.profile.label)} · ${html(row.environment.label)}</button>`).join("")}</div>
|
||||
<div class="execution-previews">${executions.map(row => `<div class="execution-preview"><div class="execution-preview-label"><strong>${html(row.profile.label)}</strong><span>${html(row.environment.label)}</span><span>${html(row.profile.provider)} · ${html(row.profile.model)}</span></div>${renderCase(row, input.expected, input.entryById)}</div>`).join("")}</div>
|
||||
</section>`;
|
||||
}).join("")}
|
||||
</section>`;
|
||||
}
|
||||
|
||||
|
||||
export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
const expected = new Set(input.expected);
|
||||
const catalog = discoverReportCatalog({
|
||||
@@ -682,7 +393,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
];
|
||||
const suiteSections = suites
|
||||
.map((suite) =>
|
||||
renderSuiteMatrix({
|
||||
renderSuiteResults({
|
||||
suiteCatalog: catalog.filter(
|
||||
(execution) => execution.suite.id === suite.id,
|
||||
),
|
||||
@@ -694,7 +405,6 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
}),
|
||||
)
|
||||
.join("");
|
||||
const historySection = renderHistory(input.history);
|
||||
const filterProfiles = [
|
||||
...new Map(
|
||||
catalog.map((execution) => [execution.profile.id, execution.profile]),
|
||||
@@ -925,40 +635,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.suite-case-toc-heading { display: flex; align-items: baseline; justify-content: space-between; gap: 16px; margin-bottom: 8px; }
|
||||
.suite-case-toc-heading h3 { margin: 0; font-size: 13px; font-weight: 600; }
|
||||
.suite-case-toc-heading > a { color: var(--muted-foreground); font-size: 11px; }
|
||||
.history-pointers { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 12px; margin-bottom: 16px; }
|
||||
.history-pointers a { padding: 14px 16px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--card); text-decoration: none; }
|
||||
.history-pointers span, .history-pointers strong { display: block; }
|
||||
.history-pointers span { color: var(--muted-foreground); font-size: 10px; letter-spacing: .06em; text-transform: uppercase; }
|
||||
.history-pointers strong { margin-top: 4px; font: 550 12px/1.4 var(--font-mono); overflow-wrap: anywhere; }
|
||||
.trend-grid { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 12px; }
|
||||
.trend-card { min-width: 0; padding: 14px 16px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--card); }
|
||||
.trend-card > span, .trend-card > strong, .trend-card > small { display: block; }
|
||||
.trend-card > span, .trend-card > small { color: var(--muted-foreground); font-size: 10px; }
|
||||
.trend-card > span { letter-spacing: .06em; text-transform: uppercase; }
|
||||
.trend-card > strong { margin-top: 4px; font: 550 17px/1.3 var(--font-mono); }
|
||||
.trend-card svg { width: 100%; height: 72px; margin: 10px 0 4px; overflow: visible; }
|
||||
.trend-card polyline { stroke: var(--foreground); stroke-width: 1.6; }
|
||||
.trend-card circle { fill: var(--background); stroke: var(--foreground); stroke-width: 1; }
|
||||
.suite-trends { margin-top: 28px; padding-top: 20px; border-top: 1px solid var(--border); }
|
||||
.suite-trends-heading { display: flex; align-items: baseline; justify-content: space-between; gap: 16px; margin-bottom: 12px; }
|
||||
.suite-trends-heading h3 { margin: 0; font: 600 14px/1.4 var(--font-mono); }
|
||||
.suite-trends-heading span { color: var(--muted-foreground); font-size: 10px; }
|
||||
.history-filters { display: grid; grid-template-columns: minmax(220px, 1fr) repeat(4, minmax(130px, 180px)) auto; align-items: end; gap: 12px; margin: 24px 0 12px; }
|
||||
.history-filters label { display: grid; gap: 5px; color: var(--muted-foreground); font-size: 10px; letter-spacing: .05em; text-transform: uppercase; }
|
||||
.history-filters input, .history-filters select { width: 100%; min-height: 38px; padding: 7px 9px; border: 1px solid var(--border-strong); border-radius: calc(var(--radius) * .8); background: var(--background); color: var(--foreground); }
|
||||
.history-filters .history-checkbox { display: flex; align-items: center; min-height: 38px; padding-bottom: 7px; white-space: nowrap; }
|
||||
.history-checkbox input { width: 16px; min-height: 16px; }
|
||||
.history-table-wrap { overflow-x: auto; border-top: 1px solid var(--border); border-bottom: 1px solid var(--border); }
|
||||
.history-table { width: 100%; min-width: 1020px; border-collapse: collapse; }
|
||||
.history-table th, .history-table td { padding: 12px; border-bottom: 1px solid var(--border); text-align: left; vertical-align: top; }
|
||||
.history-table tr:last-child td { border-bottom: 0; }
|
||||
.history-table th { background: var(--raised); color: var(--muted-foreground); font-size: 10px; letter-spacing: .05em; text-transform: uppercase; }
|
||||
.history-table td small { display: block; margin-top: 3px; color: var(--muted-foreground); font-size: 10px; }
|
||||
.history-table .status { margin-bottom: 2px; }
|
||||
.history-passed { border-color: var(--pass-border); background: var(--pass-bg); color: var(--pass-text); }
|
||||
.history-incomplete { border-color: var(--missing-border); background: var(--missing-bg); color: var(--missing-text); }
|
||||
.history-failed { border-color: var(--fail-border); background: var(--fail-bg); color: var(--fail-text); }
|
||||
.history-empty { padding: 24px 0; color: var(--muted-foreground); text-align: center; }
|
||||
|
||||
.case-context > summary, .usage > summary { width: fit-content; cursor: pointer; color: var(--foreground); font-size: 12px; font-weight: 600; }
|
||||
.case-diagnostics { margin-top: 14px; padding-top: 12px; border-top: 1px solid var(--border); }
|
||||
.case-diagnostics > summary { display: flex; align-items: baseline; flex-wrap: wrap; gap: 4px 8px; width: 100%; list-style: none; }
|
||||
@@ -1126,22 +803,20 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.table-wrap { max-height: none; overflow: visible; }
|
||||
.matrix thead th, .profile-cell, .profile-sticky { position: static; }
|
||||
}
|
||||
${reportNavigationStyles}
|
||||
</style>
|
||||
</head>
|
||||
<body id="top">
|
||||
<div class="brand-bar">
|
||||
<a class="brand-lockup" href="https://paperclip.ing" aria-label="Paperclip home">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="m16 6-8.414 8.586a2 2 0 0 0 2.829 2.829l8.414-8.586a4 4 0 1 0-5.657-5.657l-8.379 8.551a6 6 0 1 0 8.485 8.485l8.379-8.551"/></svg>
|
||||
<span>Paperclip</span>
|
||||
</a>
|
||||
<span class="brand-context">Quality engineering · Runner acceptance</span>
|
||||
</div>
|
||||
${renderReportShell({ campaignId: input.campaign?.campaignId, catalog, history: input.history })}
|
||||
<main id="overview">
|
||||
<section class="case-page-heading" data-case-page-heading hidden><p class="eyebrow">Test case</p><h1 data-case-page-title tabindex="-1"></h1><a href="?view=all" data-nav-return>← Back to campaign results</a></section>
|
||||
<section data-campaign-overview>
|
||||
<header class="report-header">
|
||||
<div>
|
||||
<p class="eyebrow">Full-stack acceptance campaign</p>
|
||||
<h1 tabindex="-1">${html(input.title)}</h1>
|
||||
<p class="lede">A browser-verified matrix of runner profiles, execution environments, and deterministic task contracts. Declared PNG screenshots and normalized results are retained with every published campaign; additional diagnostic evidence remains in the access-controlled workflow artifact.</p>
|
||||
<p class="lede">Results across runner profiles and execution environments. Declared PNG screenshots and normalized results are retained with every published campaign; additional diagnostic evidence remains in the access-controlled workflow artifact.</p>
|
||||
${input.campaign ? `<p class="report-source"><span>${html(input.generatedAt)}</span>${input.campaign.source.sha ? `<code>${html(input.campaign.source.sha.slice(0,8))}</code>` : ""}${input.campaign.source.workflowRunUrl ? `<a href="${html(input.campaign.source.workflowRunUrl)}">Source run ↗</a>` : ""}</p>` : ""}
|
||||
</div>
|
||||
<div class="report-actions">
|
||||
<div class="summary" aria-label="Campaign summary">
|
||||
@@ -1153,7 +828,8 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
<button class="gallery-launch" type="button" data-gallery-open ${screenshotCount === 0 ? "disabled" : ""}>${screenshotCount === 0 ? "Visual evidence · workflow artifact only" : `View gallery · ${screenshotCount}`}</button>
|
||||
</div>
|
||||
</header>
|
||||
${renderCampaignSuiteSummary(input.campaign?.suites ?? [], catalog, "", true)}
|
||||
${renderCampaignSuiteSummary(input.campaign?.suites ?? [], catalog, "", false)}
|
||||
<details class="campaign-billing-details"><summary>Campaign usage and spend</summary>
|
||||
<section class="billing-overview" aria-label="Campaign billing summary">
|
||||
<div class="billing-metric"><strong>${html(tokenLabel(campaignBilling.llm.inputTokens, campaignBilling.llm))}</strong><span>Input tokens</span></div>
|
||||
<div class="billing-metric"><strong>${html(tokenLabel(campaignBilling.llm.outputTokens, campaignBilling.llm))}</strong><span>Output tokens</span></div>
|
||||
@@ -1166,9 +842,10 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
<div class="billing-metric"><strong>${html(durationLabel(campaignBilling.leaseDurationMs))}</strong><span>Daytona lease time</span></div>
|
||||
<div class="billing-metric"><strong>${campaignBilling.llm.runsWithReportedCost}/${campaignBilling.llm.runCount}</strong><span>Runs provider-priced</span></div>
|
||||
<p class="billing-note">Model spend is the provider-reported subtotal; unpriced or unavailable runs are excluded, never counted as free. Daytona runtime is a public-list-price estimate from captured lease time and pinned resources, before credits, discounts, storage allowance, or invoice adjustments. Local execution has no external runtime meter.</p>
|
||||
</section></details>
|
||||
</section>
|
||||
<div class="results-heading" id="results"><h2>Results</h2><a href="#history">Campaign history</a></div>
|
||||
${renderJourneyFeatureCoverage(catalog, entryById)}
|
||||
<div class="results-heading" id="results"><h2 data-results-title>All results</h2><span data-results-description></span></div>
|
||||
<div data-feature-coverage>${renderJourneyFeatureCoverage(catalog, entryById)}</div>
|
||||
<section class="report-filters" aria-label="Filter test results">
|
||||
<label class="report-search"><span>Search tests</span><input type="search" placeholder="Test, model, provider, execution ID…" autocomplete="off" data-report-query></label>
|
||||
<button class="filter-reset" type="button" data-report-reset>Reset</button>
|
||||
@@ -1180,9 +857,9 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
<p class="filter-result" data-report-result aria-live="polite"></p>
|
||||
</section>
|
||||
${suiteSections}
|
||||
${historySection}
|
||||
<footer><span>Generated ${html(input.generatedAt)}</span><span>${catalog.length} selected executions · Declared screenshots and normalized results published</span></footer>
|
||||
</main>
|
||||
</div></div>
|
||||
<a class="back-to-top" href="#top" data-back-to-top hidden><span aria-hidden="true">↑</span>Back to top</a>
|
||||
<dialog class="gallery-dialog" data-gallery-dialog aria-labelledby="gallery-title">
|
||||
<div class="gallery-shell">
|
||||
@@ -1225,8 +902,9 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
if (event.button !== 0 || event.metaKey || event.ctrlKey || event.shiftKey || event.altKey) return;
|
||||
event.preventDefault();
|
||||
window.scrollTo({ top: 0, behavior: "instant" });
|
||||
if (location.hash !== "#top") history.pushState(null, "", "#top");
|
||||
document.querySelector(".report-header h1").focus({ preventScroll: true });
|
||||
const hash = document.querySelector('.report-shell').dataset.view === 'case' ? location.hash : '';
|
||||
history.replaceState({ ...history.state, reportScroll: 0 }, "", location.pathname + location.search + hash);
|
||||
document.querySelector("[data-case-page-heading]:not([hidden]) h1, .report-header h1").focus({ preventScroll: true });
|
||||
update();
|
||||
});
|
||||
update();
|
||||
@@ -1330,170 +1008,8 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
stage.addEventListener("pointercancel", () => { pointerStartX = null; });
|
||||
})();
|
||||
</script>
|
||||
<script>
|
||||
(() => {
|
||||
const cases = Array.from(document.querySelectorAll("[data-report-case]"));
|
||||
if (cases.length === 0) return;
|
||||
const query = document.querySelector("[data-report-query]");
|
||||
const feature = document.querySelector("[data-report-feature]");
|
||||
const profile = document.querySelector("[data-report-profile]");
|
||||
const environment = document.querySelector("[data-report-environment]");
|
||||
const suite = document.querySelector("[data-report-suite]");
|
||||
const status = document.querySelector("[data-report-status]");
|
||||
const reset = document.querySelector("[data-report-reset]");
|
||||
const result = document.querySelector("[data-report-result]");
|
||||
const galleryLaunch = document.querySelector("[data-gallery-open]");
|
||||
const caseLinks = Array.from(document.querySelectorAll("[data-report-case-link]"));
|
||||
const casesByKey = new Map();
|
||||
cases.forEach((item) => {
|
||||
const key = item.dataset.reportCaseKey;
|
||||
if (!casesByKey.has(key)) casesByKey.set(key, []);
|
||||
casesByKey.get(key).push(item);
|
||||
});
|
||||
const apply = () => {
|
||||
const search = query.value.trim().toLowerCase();
|
||||
let visible = 0;
|
||||
cases.forEach((item) => {
|
||||
const matches =
|
||||
(!search || item.dataset.reportSearch.includes(search)) &&
|
||||
(!feature.value || (feature.value === "__unmapped__" ? !item.dataset.reportFeatures : item.dataset.reportFeatures.split(" ").includes(feature.value))) &&
|
||||
(!profile.value || item.dataset.reportProfile === profile.value) &&
|
||||
(!environment.value || item.dataset.reportEnvironment === environment.value) &&
|
||||
(!suite.value || item.dataset.reportSuite === suite.value) &&
|
||||
(!status.value || item.dataset.reportStatus === status.value);
|
||||
item.hidden = !matches;
|
||||
if (matches) visible += 1;
|
||||
});
|
||||
document.querySelectorAll("[data-report-profile-row]").forEach((row) => {
|
||||
row.hidden = !row.querySelector("[data-report-case]:not([hidden])");
|
||||
});
|
||||
document.querySelectorAll(".suite-section").forEach((section) => {
|
||||
section.hidden = !section.querySelector("[data-report-case]:not([hidden])");
|
||||
});
|
||||
caseLinks.forEach((link) => {
|
||||
const variants = casesByKey.get(link.dataset.reportCaseLink);
|
||||
const matching = variants.filter((item) => !item.hidden);
|
||||
const selection = matching.length ? matching : variants;
|
||||
const counts = { passed: 0, failed: 0, incomplete: 0, missing: 0 };
|
||||
selection.forEach((item) => { counts[item.dataset.reportStatus] += 1; });
|
||||
const state = counts.failed ? "failed" : counts.incomplete ? "incomplete" : counts.missing ? "missing" : "passed";
|
||||
const label = { passed: "Passed", failed: "Failed", incomplete: "Incomplete", missing: "Missing result" }[state];
|
||||
const context = matching.length === variants.length ? "All configurations" : matching.length ? "Matching configurations" : "All configurations (none match current filters)";
|
||||
const description = label + " · " + counts.passed + "/" + selection.length + " passed"
|
||||
+ (counts.failed ? " · " + counts.failed + " failed" : "")
|
||||
+ (counts.incomplete ? " · " + counts.incomplete + " incomplete" : "")
|
||||
+ (counts.missing ? " · " + counts.missing + " missing" : "")
|
||||
+ " — " + context;
|
||||
const target = selection.find((item) => item.dataset.reportStatus === state) || selection[0];
|
||||
link.dataset.caseStatus = state;
|
||||
link.setAttribute("title", description);
|
||||
link.setAttribute("aria-description", description);
|
||||
link.setAttribute("href", "#execution-" + target.dataset.executionId);
|
||||
});
|
||||
const visibleImages = cases.reduce(
|
||||
(total, item) => total + (item.hidden ? 0 : item.querySelectorAll("[data-gallery-item]").length),
|
||||
0,
|
||||
);
|
||||
result.textContent = visible + " of " + cases.length + " tests shown";
|
||||
if (galleryLaunch) {
|
||||
galleryLaunch.disabled = visibleImages === 0;
|
||||
galleryLaunch.textContent = visibleImages === 0 ? "No visual evidence in selection" : "View gallery · " + visibleImages;
|
||||
}
|
||||
};
|
||||
const controls = [query, feature, profile, environment, suite, status];
|
||||
const clearFilters = () => {
|
||||
controls.forEach((control) => { control.value = ""; });
|
||||
};
|
||||
const revealTarget = (hash) => {
|
||||
if (!hash.startsWith("#suite-") && !hash.startsWith("#execution-")) return false;
|
||||
const target = document.getElementById(hash.slice(1));
|
||||
if (!target || !target.matches(".suite-section, [data-report-case]")) return false;
|
||||
if (target.hidden) {
|
||||
clearFilters();
|
||||
apply();
|
||||
}
|
||||
requestAnimationFrame(() => {
|
||||
target.scrollIntoView({ block: "start", behavior: "instant" });
|
||||
});
|
||||
return true;
|
||||
};
|
||||
document.querySelectorAll('a[href^="#suite-"], a[href^="#execution-"]').forEach((link) => {
|
||||
link.addEventListener("click", (event) => {
|
||||
if (event.defaultPrevented || event.button !== 0 || event.metaKey || event.ctrlKey || event.shiftKey || event.altKey) return;
|
||||
const hash = link.getAttribute("href");
|
||||
if (!revealTarget(hash)) return;
|
||||
event.preventDefault();
|
||||
if (location.hash !== hash) history.pushState(null, "", hash);
|
||||
});
|
||||
});
|
||||
window.addEventListener("hashchange", () => { revealTarget(location.hash); });
|
||||
controls.forEach((control) => {
|
||||
control.addEventListener(control === query ? "input" : "change", apply);
|
||||
});
|
||||
reset.addEventListener("click", () => {
|
||||
clearFilters();
|
||||
apply();
|
||||
query.focus();
|
||||
});
|
||||
document.querySelectorAll("[data-report-feature-choice]").forEach((button) => {
|
||||
button.addEventListener("click", () => {
|
||||
feature.value = button.dataset.reportFeatureChoice;
|
||||
apply();
|
||||
document.querySelector(".feature-coverage").open = false;
|
||||
document.getElementById("results").scrollIntoView({ block: "start" });
|
||||
});
|
||||
});
|
||||
document.addEventListener("keydown", (event) => {
|
||||
if (event.key !== "/" || event.metaKey || event.ctrlKey || event.altKey) return;
|
||||
const target = event.target;
|
||||
if (target instanceof HTMLInputElement || target instanceof HTMLSelectElement || target instanceof HTMLTextAreaElement) return;
|
||||
event.preventDefault();
|
||||
query.focus();
|
||||
});
|
||||
apply();
|
||||
revealTarget(location.hash);
|
||||
})();
|
||||
</script>
|
||||
<script>
|
||||
(() => {
|
||||
const rows = Array.from(document.querySelectorAll("[data-history-campaign]"));
|
||||
if (rows.length === 0) return;
|
||||
const query = document.querySelector("[data-history-query]");
|
||||
const suite = document.querySelector("[data-history-suite]");
|
||||
const status = document.querySelector("[data-history-status]");
|
||||
const from = document.querySelector("[data-history-from]");
|
||||
const through = document.querySelector("[data-history-through]");
|
||||
const partial = document.querySelector("[data-history-partial]");
|
||||
const empty = document.querySelector("[data-history-empty]");
|
||||
const suiteTrends = [...document.querySelectorAll("[data-history-suite-trends]")];
|
||||
const apply = () => {
|
||||
const search = query.value.trim().toLowerCase();
|
||||
let visible = 0;
|
||||
rows.forEach((row) => {
|
||||
const matches =
|
||||
(!search || row.dataset.historySearch.includes(search)) &&
|
||||
(!suite.value || row.dataset.historySuites.split(" ").includes(suite.value)) &&
|
||||
(!status.value || row.dataset.historyStatus === status.value) &&
|
||||
(!from.value || row.dataset.historyDate >= from.value) &&
|
||||
(!through.value || row.dataset.historyDate <= through.value) &&
|
||||
(partial.checked || row.dataset.historyComplete === "true");
|
||||
row.hidden = !matches;
|
||||
if (matches) visible += 1;
|
||||
});
|
||||
suiteTrends.forEach((section) => {
|
||||
section.hidden = Boolean(suite.value) && section.dataset.historySuiteTrends !== suite.value;
|
||||
});
|
||||
empty.hidden = visible !== 0;
|
||||
};
|
||||
query.addEventListener("input", apply);
|
||||
suite.addEventListener("change", apply);
|
||||
status.addEventListener("change", apply);
|
||||
from.addEventListener("change", apply);
|
||||
through.addEventListener("change", apply);
|
||||
partial.addEventListener("change", apply);
|
||||
apply();
|
||||
})();
|
||||
</script>
|
||||
<script>${reportShellScript}${reportBrowserScript}</script>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
`;
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { renderReportHistory, reportHistoryStyles, reportHistoryScript } from "./report-history.js";
|
||||
import { renderReportShell, reportNavigationStyles, reportShellScript } from "./report-navigation.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { campaignSuiteSummaryStyles, renderCampaignSuiteSummary } from "./campaign-summary.js";
|
||||
import type {
|
||||
@@ -70,34 +72,26 @@ function campaignRow(campaign: RunnerE2EHistoryCampaign) {
|
||||
)
|
||||
.join("");
|
||||
const billing = campaign.billing;
|
||||
return `<tr>
|
||||
return `<tr data-index-campaign data-index-status="${status}" data-index-search="${html([campaign.campaignId, campaign.source.ref, campaign.source.sha].join(" ").toLowerCase())}">
|
||||
<td data-label="Campaign">
|
||||
<a class="campaign-link" href="${html(campaign.publicUrl)}">${html(campaign.campaignId)}</a>
|
||||
<small>${html(date(campaign.generatedAt))} UTC</small>
|
||||
</td>
|
||||
<td data-label="Status">
|
||||
<td data-label="Result">
|
||||
<span class="status status-${status}">${status}</span>
|
||||
<small>${campaign.complete ? "Complete campaign" : "Partial campaign"}${campaign.retries > 0 ? ` · ${campaign.retries} retries` : ""}</small>
|
||||
<small>${campaign.passed}/${campaign.selected} passed · ${campaign.executed} executed</small>
|
||||
</td>
|
||||
<td data-label="Source">${sourceCell(campaign)}</td>
|
||||
<td data-label="Suites"><div class="suite-list">${suites}</div></td>
|
||||
<td data-label="Tests">
|
||||
<strong>${campaign.passed}/${campaign.selected} passed</strong>
|
||||
<small>${campaign.executed} executed · ${campaign.failed} failed${campaign.incomplete ? ` · ${campaign.incomplete} incomplete` : ""}</small>
|
||||
<td class="open-cell" data-label="Details"><a href="${html(campaign.publicUrl)}" aria-label="Open campaign ${html(campaign.campaignId)}">Open report →</a>
|
||||
<details class="campaign-row-details"><summary>Suites & billing</summary>
|
||||
<div class="suite-list">${suites}</div>
|
||||
<p>${campaign.complete ? "Complete campaign" : "Partial campaign"} · ${campaign.retries} retries · cleanup ${campaign.cleanupPassed ? "passed" : "failed"}</p>
|
||||
<p>${campaign.failed} failed${campaign.incomplete ? ` · ${campaign.incomplete} incomplete` : ""}</p>
|
||||
<p>${html(number(billing.llm.totalTokens))} tokens · ${html(number(billing.llm.inputTokens))} in · ${html(number(billing.llm.outputTokens))} out · ${html(number(billing.llm.cachedInputTokens))} cached</p>
|
||||
<p>${html(usd(billing.observedAndEstimatedCostUsd))} recorded cost · ${html(usd(billing.reportedLlmCostUsd))} LLM · ${html(usd(billing.estimatedRuntimeCostUsd))} runtime</p>
|
||||
<p>${html(duration(billing.agentRunDurationMs))} agent · ${html(duration(billing.leaseDurationMs))} Daytona lease</p>
|
||||
</details>
|
||||
</td>
|
||||
<td data-label="Tokens">
|
||||
<strong>${html(number(billing.llm.totalTokens))}</strong>
|
||||
<small>${html(number(billing.llm.inputTokens))} in · ${html(number(billing.llm.outputTokens))} out · ${html(number(billing.llm.cachedInputTokens))} cached</small>
|
||||
</td>
|
||||
<td data-label="Cost">
|
||||
<strong>${html(usd(billing.observedAndEstimatedCostUsd))}</strong>
|
||||
<small>${html(usd(billing.reportedLlmCostUsd))} LLM · ${html(usd(billing.estimatedRuntimeCostUsd))} runtime</small>
|
||||
</td>
|
||||
<td data-label="Time">
|
||||
<strong>${html(duration(billing.agentRunDurationMs))}</strong>
|
||||
<small>${html(duration(billing.leaseDurationMs))} Daytona lease</small>
|
||||
</td>
|
||||
<td class="open-cell"><a href="${html(campaign.publicUrl)}" aria-label="Open campaign ${html(campaign.campaignId)}">Open report →</a></td>
|
||||
</tr>`;
|
||||
}
|
||||
|
||||
@@ -123,7 +117,7 @@ export function renderRunnerHistoryIndex(
|
||||
const rows =
|
||||
campaigns.length > 0
|
||||
? campaigns.map(campaignRow).join("")
|
||||
: `<tr><td class="empty" colspan="9">No campaigns have been published yet.</td></tr>`;
|
||||
: `<tr><td class="empty" colspan="4">No campaigns have been published yet.</td></tr>`;
|
||||
|
||||
return `<!doctype html>
|
||||
<html lang="en">
|
||||
@@ -195,22 +189,36 @@ export function renderRunnerHistoryIndex(
|
||||
.open-cell { display:block; padding-top:14px; text-align:left; } .open-cell::before { display:none; }
|
||||
}
|
||||
@media (max-width:520px) { .summary { grid-template-columns:1fr; } .metric { border-right:0; } footer { flex-direction:column; } }
|
||||
:root { --muted-foreground:var(--muted); --card:var(--raised); --border-strong:var(--border); --font-sans:var(--font); --font-mono:var(--mono); --radius:8px; --pass-text:var(--pass); --pass-border:var(--pass); --fail-text:var(--fail); --fail-border:var(--fail); --missing-text:#b45309; --missing-bg:#fef3c7; --missing-border:#f59e0b; }
|
||||
.index-search { display:flex; justify-content:space-between; align-items:center; gap:20px; margin:28px 0 14px; }
|
||||
.index-search h2 { margin:0; font-size:15px; }
|
||||
.index-search input { width:300px; max-width:65%; min-height:36px; padding:8px 10px; background:var(--background); border:1px solid var(--border); border-radius:5px; color:var(--foreground); font:inherit; }
|
||||
.index-result { color:var(--muted); font-size:11px; }
|
||||
.report-surface h1 { font-size:32px; }
|
||||
.report-surface .lede { font-size:12px; }
|
||||
.report-surface .table-wrap { overflow-x:auto; }
|
||||
.report-surface table { table-layout:fixed; }
|
||||
.report-surface th:first-child { width:29%; }
|
||||
.report-surface th:nth-child(2) { width:20%; }
|
||||
.report-surface th:nth-child(3) { width:28%; }
|
||||
.report-surface td { overflow-wrap:anywhere; }
|
||||
.campaign-row-details { margin-top:8px; color:var(--muted); font-size:11px; text-align:left; font-weight:400; }
|
||||
.campaign-row-details summary { cursor:pointer; text-align:right; }
|
||||
.campaign-row-details p { margin:10px 0; }
|
||||
.campaign-row-details .suite-list { white-space:normal; }
|
||||
${reportNavigationStyles}
|
||||
${reportHistoryStyles}
|
||||
@media(max-width:860px) { .report-surface table { min-width:0; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="brand-bar">
|
||||
<a class="brand-lockup" href="https://paperclip.ing" aria-label="Paperclip home">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="m16 6-8.414 8.586a2 2 0 0 0 2.829 2.829l8.414-8.586a4 4 0 1 0-5.657-5.657l-8.379 8.551a6 6 0 1 0 8.485 8.485l8.379-8.551"/></svg>
|
||||
<span>Paperclip</span>
|
||||
</a>
|
||||
<span class="brand-context">Quality engineering · Runner acceptance</span>
|
||||
</div>
|
||||
${renderReportShell({ history })}
|
||||
<main>
|
||||
<header>
|
||||
<div>
|
||||
<p class="eyebrow">Historical test reporting</p>
|
||||
<h1>Runner E2E campaigns</h1>
|
||||
<p class="lede">Each row is one workflow campaign against a Paperclip revision. Open a report for its configuration matrices, matchers, per-test billing, declared screenshots, and normalized results. Additional diagnostic evidence remains in access-controlled workflow artifacts.</p>
|
||||
<h1 data-index-title>Runner E2E campaigns</h1>
|
||||
<p class="lede" data-index-description>${campaigns.length} retained campaigns. Open a report for test outcomes, per-test billing, declared screenshots, and normalized results.</p>
|
||||
</div>
|
||||
<div class="summary" aria-label="History summary">
|
||||
<div class="metric"><strong>${campaigns.length}</strong><span>Campaigns</span></div>
|
||||
@@ -218,6 +226,7 @@ export function renderRunnerHistoryIndex(
|
||||
<div class="metric"><strong>${html(usd(totalCost))}</strong><span>Recorded cost</span></div>
|
||||
</div>
|
||||
</header>
|
||||
<section data-index-overview>
|
||||
${latest ? `<section class="latest-campaign" aria-label="Latest campaign summary">
|
||||
<div class="latest-campaign-heading"><h2>Latest campaign</h2><a class="campaign-link" href="${html(latest.publicUrl)}">${html(latest.campaignId)}</a></div>
|
||||
<dl class="latest-metrics">
|
||||
@@ -231,14 +240,49 @@ export function renderRunnerHistoryIndex(
|
||||
${latest ? `<a href="${html(latest.publicUrl)}">Latest run · ${html(latest.campaignId)}</a>` : ""}
|
||||
${latestGreen ? `<a href="${html(latestGreen.publicUrl)}">Latest complete green · ${html(latestGreen.campaignId)}</a>` : ""}
|
||||
</nav>
|
||||
</section>
|
||||
<div class="index-search"><h2 data-index-list-title>Recent campaigns</h2><input type="search" data-index-query aria-label="Search campaigns" placeholder="Find a campaign, revision, or branch…"></div>
|
||||
<p class="index-result" data-index-result aria-live="polite"></p>
|
||||
<div class="table-wrap">
|
||||
<table>
|
||||
<thead><tr><th>Campaign</th><th>Status</th><th>Source</th><th>Suites</th><th>Tests</th><th>Tokens</th><th>Cost</th><th>Agent time</th><th></th></tr></thead>
|
||||
<thead><tr><th>Campaign</th><th>Result</th><th>Source</th><th></th></tr></thead>
|
||||
<tbody>${rows}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<details data-index-trends hidden><summary>Trends and detailed history</summary>${renderReportHistory(history)}</details>
|
||||
<footer><span>Updated ${html(date(history.updatedAt))} UTC</span><span>Immutable campaign reports · Declared screenshots and normalized results</span></footer>
|
||||
</main>
|
||||
</main></div></div>
|
||||
<script>${reportShellScript}${reportHistoryScript}
|
||||
(() => {
|
||||
const shell = document.querySelector('.report-shell');
|
||||
const rows = [...document.querySelectorAll('[data-index-campaign]')];
|
||||
const query = document.querySelector('[data-index-query]');
|
||||
const crumbs = document.querySelector('[data-nav-breadcrumbs]');
|
||||
const originalDescription = document.querySelector('[data-index-description]').textContent;
|
||||
const apply = () => {
|
||||
const params = new URLSearchParams(location.search);
|
||||
const view = ['history','attention'].includes(params.get('view')) ? params.get('view') : 'overview';
|
||||
const title = {overview:'Runner E2E campaigns',history:'Campaign history',attention:'Needs attention'}[view];
|
||||
shell.dataset.view = view;
|
||||
document.querySelector('[data-index-title]').textContent = title;
|
||||
document.querySelector('[data-index-overview]').hidden = view !== 'overview';
|
||||
document.querySelector('[data-index-trends]').hidden = view !== 'history';
|
||||
document.querySelector('[data-index-description]').textContent = view === 'attention' ? 'Campaigns with failed or incomplete executions, missing results, or cleanup failures.' : originalDescription;
|
||||
document.querySelector('[data-index-list-title]').textContent = view === 'overview' ? 'Recent campaigns' : 'Campaigns';
|
||||
crumbs.replaceChildren();
|
||||
const root = document.createElement('a'); root.href = '?view=history'; root.textContent = 'Campaigns'; root.dataset.navView = 'history'; crumbs.append(root);
|
||||
const separator = document.createElement('span');separator.setAttribute('aria-hidden','true');separator.textContent='/';crumbs.append(separator);
|
||||
const current = document.createElement('span');current.setAttribute('aria-current','page');current.textContent = view === 'overview' ? 'Overview' : title;crumbs.append(current);
|
||||
document.querySelectorAll('[data-nav-view]').forEach(link => { if(link.dataset.navView===view)link.setAttribute('aria-current','page');else link.removeAttribute('aria-current'); });
|
||||
const search=query.value.trim().toLowerCase(); let visible=0,matching=0;
|
||||
rows.forEach(row => { const match=(!search||row.dataset.indexSearch.includes(search))&&(view!=='attention'||row.dataset.indexStatus!=='passed');if(match)matching++;row.hidden=!match||(view==='overview'&&!search&&visible>=10);if(!row.hidden)visible++; });
|
||||
document.querySelector('[data-index-result]').textContent=visible+' of '+matching+' campaigns'+(view==='overview'&&!search&&matching>10?' · Open Campaign history for all retained reports':'');
|
||||
};
|
||||
document.addEventListener('click',event=>{ const link=event.target.closest('[data-nav-view]');if(!link||event.defaultPrevented||event.button!==0||event.metaKey||event.ctrlKey||event.shiftKey||event.altKey)return;event.preventDefault();const url=new URL(location.href);url.searchParams.set('view',link.dataset.navView);url.searchParams.delete('q');history.pushState(null,'',url);query.value='';apply();window.scrollTo({top:0,behavior:'instant'}); });
|
||||
query.addEventListener('input',()=>{const url=new URL(location.href);if(query.value)url.searchParams.set('q',query.value);else url.searchParams.delete('q');history.replaceState(null,'',url);apply();});
|
||||
const read=()=>{query.value=new URLSearchParams(location.search).get('q')||'';apply();};window.addEventListener('popstate',read);read();
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>`;
|
||||
}
|
||||
@@ -137,7 +137,7 @@ describe("runner E2E campaign history", () => {
|
||||
expect(history.latestGreenCampaignId).toBeNull();
|
||||
expect(history.latestGreenBySuite["first-task"]).toBeUndefined();
|
||||
expect(renderRunnerHistoryIndex(history)).toContain("1 incomplete");
|
||||
expect(renderRunnerE2EDashboard({ title: "History", generatedAt: incomplete.finishedAt, expected: [], catalog: [], entries: [], history })).toContain('data-history-status="incomplete"');
|
||||
expect(renderRunnerHistoryIndex(history)).toContain('data-history-status="incomplete"');
|
||||
});
|
||||
|
||||
it("shows unknown campaign costs without plotting zero dollars", () => {
|
||||
@@ -147,9 +147,9 @@ describe("runner E2E campaign history", () => {
|
||||
const record = { ...campaignHistoryRecord(campaign, "https://example.test"), complete: true };
|
||||
const history = mergeRunnerHistory(null, record);
|
||||
expect(renderRunnerHistoryIndex(history)).toContain("Unknown");
|
||||
const dashboard = renderRunnerE2EDashboard({ title: "History", generatedAt: campaign.generatedAt, expected: [], catalog: [], entries: [], history });
|
||||
expect(dashboard).toContain("Unknown measurements are not plotted");
|
||||
expect(dashboard).not.toContain("NaN");
|
||||
const index = renderRunnerHistoryIndex(history);
|
||||
expect(index).toContain("Unknown measurements are not plotted");
|
||||
expect(index).not.toContain("NaN");
|
||||
});
|
||||
|
||||
it("does not turn a clean-evidence behavior failure into a pass when regenerating", async () => {
|
||||
@@ -274,22 +274,23 @@ describe("runner E2E campaign history", () => {
|
||||
errors: campaignResult.status === "passed" ? [] : ["failed"],
|
||||
})),
|
||||
});
|
||||
expect(dashboard).toContain("Campaign trends");
|
||||
expect(dashboard).toContain("data-history-from");
|
||||
expect(dashboard).toContain("data-history-through");
|
||||
expect(dashboard).toContain(
|
||||
expect(dashboard).not.toContain('class="history-section"');
|
||||
const index = renderRunnerHistoryIndex(history);
|
||||
expect(index).toContain("Campaign trends");
|
||||
expect(index).toContain("data-history-from");
|
||||
expect(index).toContain("data-history-through");
|
||||
expect(index).toContain(
|
||||
'data-history-suite-trends="core-compatibility"',
|
||||
);
|
||||
expect(dashboard).toContain(
|
||||
expect(index).toContain(
|
||||
'data-history-suite-trends="openrouter-model-breadth"',
|
||||
);
|
||||
expect(dashboard).toContain(
|
||||
expect(index).toContain(
|
||||
'data-history-suite-trends="local-session-integrity"',
|
||||
);
|
||||
expect(dashboard).toContain("Suite pass rate");
|
||||
expect(dashboard).toContain("lines break at definition changes");
|
||||
expect(dashboard).toContain("cleanup passed");
|
||||
const index = renderRunnerHistoryIndex(history);
|
||||
expect(index).toContain("Suite pass rate");
|
||||
expect(index).toContain("lines break at definition changes");
|
||||
expect(index).toContain("cleanup passed");
|
||||
expect(index).toContain("Runner E2E campaigns");
|
||||
expect(index).toContain("complete-green");
|
||||
expect(index).toContain("complete-red");
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
export const paperclipReportLogo = `<svg class="report-brand-logo" viewBox="22.5 22.5 121 27" fill="currentColor" aria-hidden="true" focusable="false">
|
||||
<path d="M131.15 48.4902V31.9902H133.922V33.9775H134.094C134.237 33.6911 134.441 33.3831 134.706 33.0537C134.971 32.7171 135.329 32.4307 135.78 32.1943C136.239 31.958 136.822 31.8398 137.531 31.8398C138.455 31.8398 139.289 32.0762 140.034 32.5488C140.779 33.0215 141.37 33.7161 141.807 34.6328C142.251 35.5495 142.473 36.6774 142.473 38.0166C142.473 39.3343 142.258 40.4551 141.828 41.3789C141.398 42.3027 140.808 43.0081 140.056 43.4951C139.311 43.9749 138.466 44.2148 137.521 44.2148C136.84 44.2148 136.271 44.1003 135.812 43.8711C135.361 43.6348 134.996 43.3555 134.717 43.0332C134.445 42.7038 134.237 42.3923 134.094 42.0986H133.976V48.4902H131.15ZM136.747 41.8945C137.37 41.8945 137.889 41.7262 138.305 41.3896C138.727 41.0531 139.046 40.5911 139.261 40.0039C139.476 39.4167 139.583 38.7507 139.583 38.0059C139.583 37.2611 139.476 36.5986 139.261 36.0186C139.046 35.4385 138.731 34.9837 138.315 34.6543C137.9 34.3249 137.377 34.1602 136.747 34.1602C136.138 34.1602 135.623 34.3177 135.2 34.6328C134.785 34.9479 134.466 35.3955 134.244 35.9756C134.029 36.5485 133.922 37.2253 133.922 38.0059C133.922 38.7793 134.029 39.4596 134.244 40.0469C134.466 40.627 134.788 41.0817 135.211 41.4111C135.633 41.7334 136.146 41.8945 136.747 41.8945Z" />
|
||||
<path d="M126.451 44V31.9902H129.276V44H126.451ZM127.869 30.3037C127.417 30.3037 127.031 30.1533 126.708 29.8525C126.386 29.5518 126.225 29.1901 126.225 28.7676C126.225 28.3379 126.386 27.9727 126.708 27.6719C127.031 27.3711 127.417 27.2207 127.869 27.2207C128.313 27.2207 128.696 27.3711 129.018 27.6719C129.34 27.9727 129.501 28.3343 129.501 28.7568C129.501 29.1865 129.34 29.5518 129.018 29.8525C128.696 30.1533 128.313 30.3037 127.869 30.3037Z" />
|
||||
<path d="M124.576 27.9941V44H121.75V27.9941H124.576Z" />
|
||||
<path d="M115.3 44.2363C114.104 44.2363 113.076 43.9749 112.217 43.4521C111.357 42.9222 110.699 42.1917 110.24 41.2607C109.782 40.3298 109.553 39.2591 109.553 38.0488C109.553 36.8242 109.785 35.7464 110.251 34.8154C110.716 33.8844 111.379 33.1576 112.238 32.6348C113.098 32.1048 114.111 31.8398 115.278 31.8398C116.259 31.8398 117.126 32.0189 117.878 32.377C118.63 32.735 119.228 33.2399 119.672 33.8916C120.123 34.5361 120.381 35.2917 120.445 36.1582H117.738C117.667 35.7715 117.527 35.4242 117.319 35.1162C117.112 34.8083 116.84 34.5648 116.503 34.3857C116.174 34.2067 115.78 34.1172 115.321 34.1172C114.741 34.1172 114.233 34.2747 113.796 34.5898C113.366 34.8978 113.03 35.3418 112.786 35.9219C112.543 36.502 112.421 37.1966 112.421 38.0059C112.421 38.8223 112.539 39.5277 112.775 40.1221C113.012 40.7093 113.348 41.1641 113.785 41.4863C114.222 41.8014 114.734 41.959 115.321 41.959C115.944 41.959 116.471 41.7764 116.9 41.4111C117.337 41.0459 117.617 40.541 117.738 39.8965H120.445C120.381 40.7487 120.13 41.5007 119.693 42.1523C119.257 42.804 118.666 43.3161 117.921 43.6885C117.176 44.0537 116.302 44.2363 115.3 44.2363Z" />
|
||||
<path d="M102.876 44V31.9902H105.615V34.0098H105.744C105.959 33.3079 106.335 32.7708 106.872 32.3984C107.409 32.0189 108.021 31.8291 108.709 31.8291C108.859 31.8291 109.031 31.8363 109.225 31.8506C109.425 31.8649 109.59 31.8828 109.719 31.9043V34.5039C109.597 34.4609 109.407 34.4251 109.149 34.3965C108.892 34.3678 108.645 34.3535 108.408 34.3535C107.885 34.3535 107.42 34.4645 107.012 34.6865C106.604 34.9085 106.281 35.2129 106.045 35.5996C105.816 35.9863 105.701 36.4375 105.701 36.9531V44H102.876Z" />
|
||||
<path d="M96.2319 44.2363C95.0288 44.2363 93.9904 43.9857 93.1167 43.4844C92.243 42.9759 91.5698 42.2598 91.0972 41.3359C90.6317 40.4121 90.3989 39.3236 90.3989 38.0703C90.3989 36.8314 90.6317 35.7464 91.0972 34.8154C91.5627 33.8773 92.2179 33.1468 93.063 32.624C93.908 32.1012 94.9035 31.8398 96.0493 31.8398C96.7869 31.8398 97.4852 31.958 98.144 32.1943C98.8101 32.4307 99.3973 32.7959 99.9058 33.29C100.421 33.7842 100.826 34.4144 101.12 35.1807C101.413 35.9398 101.56 36.8457 101.56 37.8984V38.7793H91.7202V36.8779H100.153L98.8315 37.4365C98.8315 36.7562 98.7277 36.1654 98.52 35.6641C98.3123 35.1556 98.0044 34.7581 97.5962 34.4717C97.188 34.1852 96.6795 34.042 96.0708 34.042C95.4621 34.042 94.9429 34.1852 94.5132 34.4717C94.0835 34.7581 93.7576 35.1449 93.5356 35.6318C93.3136 36.1117 93.2026 36.6488 93.2026 37.2432V38.5645C93.2026 39.2949 93.328 39.9215 93.5786 40.4443C93.8364 40.96 94.1945 41.3538 94.6528 41.626C95.1183 41.8981 95.659 42.0342 96.2749 42.0342C96.6759 42.0342 97.0412 41.9769 97.3706 41.8623C97.7072 41.7477 97.9937 41.5758 98.23 41.3467C98.4735 41.1104 98.6597 40.8203 98.7886 40.4766L101.431 40.7881C101.259 41.4827 100.941 42.0915 100.475 42.6143C100.017 43.1299 99.4259 43.5309 98.7026 43.8174C97.9865 44.0967 97.1629 44.2363 96.2319 44.2363Z" />
|
||||
<path d="M78.2437 48.4902V31.9902H81.0151V33.9775H81.187C81.3302 33.6911 81.5343 33.3831 81.7993 33.0537C82.0643 32.7171 82.4224 32.4307 82.8735 32.1943C83.3319 31.958 83.9155 31.8398 84.6245 31.8398C85.5483 31.8398 86.3826 32.0762 87.1274 32.5488C87.8722 33.0215 88.4631 33.7161 88.8999 34.6328C89.3439 35.5495 89.5659 36.6774 89.5659 38.0166C89.5659 39.3343 89.3511 40.4551 88.9214 41.3789C88.4917 42.3027 87.9009 43.0081 87.1489 43.4951C86.4041 43.9749 85.5591 44.2148 84.6138 44.2148C83.9334 44.2148 83.3641 44.1003 82.9058 43.8711C82.4546 43.6348 82.0894 43.3555 81.8101 43.0332C81.5379 42.7038 81.3302 42.3923 81.187 42.0986H81.0688V48.4902H78.2437ZM83.8403 41.8945C84.4634 41.8945 84.9826 41.7262 85.3979 41.3896C85.8205 41.0531 86.1392 40.5911 86.354 40.0039C86.5688 39.4167 86.6763 38.7507 86.6763 38.0059C86.6763 37.2611 86.5688 36.5986 86.354 36.0186C86.1392 35.4385 85.8241 34.9837 85.4087 34.6543C84.9933 34.3249 84.4705 34.1602 83.8403 34.1602C83.2316 34.1602 82.716 34.3177 82.2935 34.6328C81.8781 34.9479 81.5594 35.3955 81.3374 35.9756C81.1226 36.5485 81.0151 37.2253 81.0151 38.0059C81.0151 38.7793 81.1226 39.4596 81.3374 40.0469C81.5594 40.627 81.8817 41.0817 82.3042 41.4111C82.7267 41.7334 83.2388 41.8945 83.8403 41.8945Z" />
|
||||
<path d="M70.042 44.2471C69.2757 44.2471 68.5882 44.111 67.9795 43.8389C67.3708 43.5596 66.891 43.1514 66.54 42.6143C66.1891 42.0771 66.0137 41.4147 66.0137 40.627C66.0137 39.9538 66.139 39.3952 66.3896 38.9512C66.6403 38.5072 66.9805 38.1527 67.4102 37.8877C67.847 37.6227 68.3376 37.4222 68.8818 37.2861C69.4261 37.1429 69.9883 37.0426 70.5684 36.9854C71.2773 36.9066 71.8503 36.8385 72.2871 36.7812C72.724 36.724 73.0391 36.6309 73.2324 36.502C73.4329 36.373 73.5332 36.1725 73.5332 35.9004V35.8467C73.5332 35.2666 73.3613 34.8154 73.0176 34.4932C72.6738 34.1709 72.1725 34.0098 71.5137 34.0098C70.819 34.0098 70.2676 34.1602 69.8594 34.4609C69.4583 34.7617 69.1862 35.1162 69.043 35.5244L66.4111 35.1484C66.6188 34.418 66.9626 33.8092 67.4424 33.3223C67.9222 32.8281 68.5059 32.4593 69.1934 32.2158C69.888 31.9652 70.6543 31.8398 71.4922 31.8398C72.0651 31.8398 72.638 31.9079 73.2109 32.0439C73.7839 32.1729 74.3102 32.3949 74.79 32.71C75.2699 33.0251 75.653 33.4476 75.9395 33.9775C76.2259 34.5075 76.3691 35.1699 76.3691 35.9648V44H73.6514V42.3457H73.5654C73.3936 42.6751 73.1501 42.9867 72.835 43.2803C72.527 43.5667 72.1403 43.7995 71.6748 43.9785C71.2165 44.1576 70.6722 44.2471 70.042 44.2471ZM70.7617 42.1631C71.3346 42.1631 71.8288 42.0521 72.2441 41.8301C72.6595 41.6009 72.9818 41.3001 73.2109 40.9277C73.4401 40.5482 73.5547 40.1364 73.5547 39.6924V38.2852C73.4616 38.3568 73.3112 38.4248 73.1035 38.4893C72.8958 38.5465 72.6595 38.6003 72.3945 38.6504C72.1296 38.7005 71.8682 38.7435 71.6104 38.7793C71.3597 38.8151 71.1413 38.8438 70.9551 38.8652C70.5326 38.9225 70.153 39.0192 69.8164 39.1553C69.487 39.2842 69.2256 39.4632 69.0322 39.6924C68.8389 39.9215 68.7422 40.2188 68.7422 40.584C68.7422 41.1068 68.932 41.5007 69.3115 41.7656C69.6911 42.0306 70.1745 42.1631 70.7617 42.1631Z" />
|
||||
<path d="M53.7939 44V27.9941H59.7988C61.0306 27.9941 62.0618 28.2233 62.8926 28.6816C63.7305 29.14 64.3643 29.7702 64.7939 30.5723C65.2236 31.3743 65.4385 32.2839 65.4385 33.3008C65.4385 34.3249 65.2201 35.238 64.7832 36.04C64.3535 36.835 63.7161 37.4616 62.8711 37.9199C62.026 38.3711 60.9876 38.5967 59.7559 38.5967H55.7812V36.2119H59.3584C60.0817 36.2119 60.6725 36.0902 61.1309 35.8467C61.5964 35.596 61.9365 35.2523 62.1514 34.8154C62.3734 34.3714 62.4844 33.8665 62.4844 33.3008C62.4844 32.7279 62.3734 32.2266 62.1514 31.7969C61.9365 31.36 61.5964 31.0234 61.1309 30.7871C60.6725 30.5436 60.0781 30.4219 59.3477 30.4219H56.6943V44H53.7939Z" />
|
||||
<path d="M46.2611 33.6556L34.7307 44.6408C32.7445 46.5331 29.5205 46.5323 27.5303 44.6361C25.5404 42.74 25.54 39.669 27.5261 37.7768L39.0565 26.7916C40.2474 25.657 42.1811 25.6567 43.3752 26.7943C44.5693 27.9319 44.5689 29.7742 43.378 30.9088L31.849 41.8954C31.4528 42.2728 30.8077 42.273 30.4092 41.8933C30.011 41.5136 30.0117 40.8996 30.4078 40.5223L41.9382 29.5371C42.336 29.1581 42.336 28.5437 41.9382 28.1647C41.5404 27.786 40.8954 27.7858 40.4977 28.1647L28.9673 39.1499C27.7742 40.2866 27.7764 42.1295 28.9687 43.2657C30.1613 44.402 32.0963 44.4046 33.2895 43.2678L44.8192 32.2805C46.8068 30.3865 46.8038 27.3159 44.8157 25.4219C42.8275 23.5277 39.6039 23.5253 37.616 25.4192L26.0856 36.4044C23.3024 39.056 23.3058 43.3558 26.0898 46.0085C28.8741 48.6612 33.388 48.6649 36.1712 46.0132L47.7017 35.028C48.0994 34.649 48.0994 34.0346 47.7017 33.6556C47.3039 33.2766 46.6589 33.2766 46.2611 33.6556Z" />
|
||||
</svg>`;
|
||||
@@ -0,0 +1,176 @@
|
||||
export const reportBrowserScript = String.raw`
|
||||
(() => {
|
||||
const shell = document.querySelector('[data-report-scope="campaign"]');
|
||||
if (!shell) return;
|
||||
const cards = [...document.querySelectorAll('[data-report-case]')];
|
||||
const sections = [...document.querySelectorAll('[data-report-suite-section]')];
|
||||
const journeys = [...document.querySelectorAll('[data-report-journey]')];
|
||||
const controls = ['query','feature','profile','environment','suite','status'].map(name => document.querySelector('[data-report-' + name + ']'));
|
||||
const [query,feature,profile,environment,suite,status] = controls;
|
||||
const paramNames = ['q','feature','profile','environment','suite','status'];
|
||||
const byKey = new Map();
|
||||
cards.forEach(card => { const key = card.dataset.reportCaseKey; if (!byKey.has(key)) byKey.set(key, []); byKey.get(key).push(card); });
|
||||
const crumbs = document.querySelector('[data-nav-breadcrumbs]');
|
||||
const historyHref = crumbs.querySelector('a').getAttribute('href');
|
||||
const heading = document.querySelector('[data-case-page-heading]');
|
||||
const result = document.querySelector('[data-report-result]');
|
||||
let view = 'all', caseKey = '', configuration = '', lastReport = null, restoring = false;
|
||||
history.scrollRestoration = 'manual';
|
||||
const normalClick = event => event.button === 0 && !event.metaKey && !event.ctrlKey && !event.shiftKey && !event.altKey;
|
||||
const savePosition = () => { if (!restoring) history.replaceState({ ...history.state, reportScroll: window.scrollY }, '', location.href); };
|
||||
const severity = rows => rows.some(row => row.dataset.reportStatus === 'failed') ? 'failed' : rows.some(row => row.dataset.reportStatus === 'incomplete') ? 'incomplete' : rows.some(row => row.dataset.reportStatus === 'missing') ? 'missing' : 'passed';
|
||||
const clearFilters = () => controls.forEach(control => { control.value = ''; });
|
||||
const routeUrl = (nextView, key = '', hash = '') => {
|
||||
const url = new URL(location.href);
|
||||
url.searchParams.set('view', nextView);
|
||||
if (key) url.searchParams.set('case', key); else url.searchParams.delete('case');
|
||||
controls.forEach((control, i) => { if(control.value) url.searchParams.set(paramNames[i], control.value); else url.searchParams.delete(paramNames[i]); });
|
||||
url.hash = hash;
|
||||
return url.pathname + url.search + url.hash;
|
||||
};
|
||||
const readRoute = () => {
|
||||
const params = new URLSearchParams(location.search);
|
||||
view = ['all','attention','case'].includes(params.get('view')) ? params.get('view') : 'all';
|
||||
caseKey = params.get('case') || '';
|
||||
configuration = location.hash.startsWith('#execution-') ? decodeURIComponent(location.hash.slice(11)) : '';
|
||||
controls.forEach((control,i) => { control.value = params.get(paramNames[i]) || ''; });
|
||||
if(location.hash === '#history') { location.replace(historyHref); return false; }
|
||||
return true;
|
||||
};
|
||||
const appendCrumb = (label, href, attributes = {}) => {
|
||||
if(crumbs.childNodes.length) { const separator = document.createElement('span'); separator.setAttribute('aria-hidden','true'); separator.textContent = '/'; crumbs.append(separator); }
|
||||
const node = document.createElement(href ? 'a' : 'span'); node.textContent = label;
|
||||
if(href) node.setAttribute('href', href); else node.setAttribute('aria-current','page');
|
||||
Object.entries(attributes).forEach(([name,value]) => node.setAttribute(name,value)); crumbs.append(node);
|
||||
};
|
||||
const setBreadcrumbs = selected => {
|
||||
crumbs.replaceChildren();
|
||||
appendCrumb('Campaigns', historyHref);
|
||||
appendCrumb(shell.dataset.campaignLabel, lastReport?.url || routeUrl('all'), {'data-nav-return':''});
|
||||
if(view === 'case' && selected) {
|
||||
const section = selected.closest('[data-report-suite-section]');
|
||||
appendCrumb(section.dataset.suiteLabel, routeUrl('all','','#' + section.id), {'data-nav-suite-link':selected.dataset.reportSuite});
|
||||
appendCrumb(selected.dataset.reportLabel);
|
||||
} else appendCrumb(view === 'attention' ? 'Needs attention' : view === 'case' ? 'Case unavailable' : 'All results');
|
||||
};
|
||||
const apply = () => {
|
||||
const search = query.value.trim().toLowerCase();
|
||||
const matchesFilters = card =>
|
||||
(!search || card.dataset.reportSearch.includes(search)) &&
|
||||
(!feature.value || (feature.value === '__unmapped__' ? !card.dataset.reportFeatures : card.dataset.reportFeatures.split(' ').includes(feature.value))) &&
|
||||
(!profile.value || card.dataset.reportProfile === profile.value) &&
|
||||
(!environment.value || card.dataset.reportEnvironment === environment.value) &&
|
||||
(!suite.value || card.dataset.reportSuite === suite.value) &&
|
||||
(!status.value || card.dataset.reportStatus === status.value);
|
||||
const variants = byKey.get(caseKey) || [];
|
||||
const matchingVariants = variants.filter(matchesFilters);
|
||||
const candidates = matchingVariants.length ? matchingVariants : variants;
|
||||
let selected = variants.find(row => row.dataset.executionId === configuration);
|
||||
if(!selected) selected = candidates.find(row => row.dataset.reportStatus === severity(candidates)) || candidates[0];
|
||||
configuration = selected?.dataset.executionId || '';
|
||||
let visible = 0;
|
||||
cards.forEach(card => {
|
||||
const matches = view === 'case' ? card === selected :
|
||||
(view !== 'attention' || card.dataset.reportStatus !== 'passed') && matchesFilters(card);
|
||||
card.hidden = !matches;
|
||||
card.closest('.execution-preview').hidden = !matches;
|
||||
if(matches) visible++;
|
||||
});
|
||||
journeys.forEach(journey => {
|
||||
journey.hidden = !journey.querySelector('[data-report-case]:not([hidden])');
|
||||
journey.querySelector('.case-config-options').hidden = view !== 'case';
|
||||
});
|
||||
sections.forEach(section => { section.hidden = !section.querySelector('[data-report-case]:not([hidden])'); });
|
||||
document.querySelectorAll('[data-nav-configuration]').forEach(button => {
|
||||
const card = document.getElementById('execution-' + button.dataset.navConfiguration);
|
||||
button.setAttribute('aria-pressed', String(card === selected));
|
||||
button.dataset.caseStatus = card.dataset.reportStatus;
|
||||
button.setAttribute('aria-description', card.dataset.reportStatus);
|
||||
});
|
||||
document.querySelectorAll('[data-report-case-link]').forEach(link => {
|
||||
const variants = byKey.get(link.dataset.reportCaseLink) || [];
|
||||
const matching = variants.filter(matchesFilters);
|
||||
const rows = matching.length ? matching : variants;
|
||||
if(!rows.length) return;
|
||||
const state = severity(rows), target = rows.find(row => row.dataset.reportStatus === state) || rows[0];
|
||||
const passed = rows.filter(row => row.dataset.reportStatus === 'passed').length;
|
||||
const description = state.replaceAll('-', ' ') + ' · ' + passed + '/' + rows.length + ' configurations passed' + (matching.length ? '' : ' · no matching configurations');
|
||||
link.dataset.caseStatus = state; link.title = description; link.setAttribute('aria-description', description);
|
||||
link.setAttribute('href', link.hasAttribute('data-nav-case') ? routeUrl('case', link.dataset.navCase, '#execution-' + target.dataset.executionId) : '#execution-' + target.dataset.executionId);
|
||||
});
|
||||
shell.dataset.view = view;
|
||||
document.querySelector('[data-campaign-overview]').hidden = view !== 'all';
|
||||
document.querySelector('[data-feature-coverage]').hidden = view === 'case';
|
||||
document.querySelector('.report-filters').hidden = view === 'case';
|
||||
document.querySelector('.results-heading').hidden = view === 'case';
|
||||
document.querySelector('[data-results-title]').textContent = view === 'attention' ? 'Needs attention' : 'All results';
|
||||
document.querySelector('[data-results-description]').textContent = view === 'attention' ? 'Failed, incomplete, and missing executions in this campaign' : '';
|
||||
heading.hidden = view !== 'case';
|
||||
document.querySelector('[data-case-page-title]').textContent = selected?.dataset.reportLabel || 'This case was not selected in this campaign';
|
||||
document.querySelector('[data-nav-return]').setAttribute('href', lastReport?.url || routeUrl('all'));
|
||||
document.querySelectorAll('[data-nav-view]').forEach(link => { if(link.dataset.navView === view) link.setAttribute('aria-current','page'); else link.removeAttribute('aria-current'); });
|
||||
document.querySelectorAll('[data-nav-suite-link]').forEach(link => link.setAttribute('href',routeUrl('all','','#suite-' + link.dataset.navSuiteLink)));
|
||||
document.querySelectorAll('[data-nav-case]').forEach(link => { if(view === 'case' && link.dataset.navCase === caseKey) link.setAttribute('aria-current','page'); else link.removeAttribute('aria-current'); });
|
||||
const selectedSuite = selected?.dataset.reportSuite;
|
||||
if(view === 'case') document.querySelectorAll('[data-nav-suite]').forEach(details => { if(details.dataset.navSuite === selectedSuite) details.open = true; });
|
||||
result.textContent = visible + ' of ' + cards.length + ' executions shown' + (!visible ? ' — No executions match this selection.' : '');
|
||||
const images = cards.reduce((n,card) => n + (card.hidden ? 0 : card.querySelectorAll('[data-gallery-item]').length),0);
|
||||
const launch = document.querySelector('[data-gallery-open]');
|
||||
launch.disabled = images === 0; launch.textContent = images ? 'View gallery · ' + images : 'No visual evidence in selection';
|
||||
setBreadcrumbs(selected);
|
||||
measureHeader();
|
||||
};
|
||||
const measureHeader = () => document.documentElement.style.setProperty('--navigation-sticky-offset',document.querySelector('.report-topbar').offsetHeight + 'px');
|
||||
window.addEventListener('resize',measureHeader);
|
||||
const navigate = (url, scroll) => {
|
||||
savePosition(); history.pushState({reportScroll:scroll || 0},'',url);
|
||||
readRoute(); restoring = true; apply();
|
||||
if(typeof scroll === 'number') window.scrollTo({top:scroll,behavior:'instant'});
|
||||
else if(location.hash) revealTarget(location.hash);
|
||||
else window.scrollTo({top:0,behavior:'instant'});
|
||||
restoring = false;
|
||||
};
|
||||
const revealTarget = hash => {
|
||||
const id = decodeURIComponent(hash.slice(1));
|
||||
const target = document.getElementById(id);
|
||||
if(!target || !target.matches('.suite-section,[data-report-case],.report-journey,#overview,#results,#top')) return;
|
||||
if(target.hidden || target.closest('[hidden]')) {
|
||||
clearFilters(); view = 'all'; caseKey = ''; apply();
|
||||
history.replaceState(history.state,'',routeUrl('all','',hash));
|
||||
}
|
||||
target.scrollIntoView({block:'start',behavior:'instant'});
|
||||
};
|
||||
document.addEventListener('click', event => {
|
||||
if(event.defaultPrevented || !normalClick(event)) return;
|
||||
const link = event.target.closest('a');
|
||||
if(link?.hasAttribute('data-nav-return')) {
|
||||
event.preventDefault(); navigate(lastReport?.url || routeUrl('all'),lastReport?.scroll || 0); return;
|
||||
}
|
||||
if(link?.hasAttribute('data-nav-case')) {
|
||||
event.preventDefault();
|
||||
if(view !== 'case') lastReport = {url:location.pathname + location.search + location.hash,scroll:window.scrollY};
|
||||
const hash = new URL(link.href).hash;
|
||||
navigate(routeUrl('case',link.dataset.navCase,hash),0); return;
|
||||
}
|
||||
if(link?.hasAttribute('data-nav-view')) { event.preventDefault(); navigate(routeUrl(link.dataset.navView)); return; }
|
||||
if(link?.hasAttribute('data-nav-suite-link')) {
|
||||
event.preventDefault(); const id = link.dataset.navSuiteLink;
|
||||
document.querySelectorAll('[data-nav-suite]').forEach(details => {if(details.dataset.navSuite === id)details.open = true;});
|
||||
navigate(routeUrl('all','','#suite-' + id)); return;
|
||||
}
|
||||
if(link?.getAttribute('href')?.startsWith('#') && /^#(suite-|execution-|journey-|overview|results)/.test(link.getAttribute('href'))) {
|
||||
event.preventDefault(); navigate(routeUrl('all','',link.getAttribute('href'))); return;
|
||||
}
|
||||
const config = event.target.closest('[data-nav-configuration]');
|
||||
if(config) { configuration = config.dataset.navConfiguration; history.replaceState(history.state,'',routeUrl('case',caseKey,'#execution-' + configuration)); apply(); }
|
||||
});
|
||||
controls.forEach(control => control.addEventListener(control === query ? 'input' : 'change', () => { history.replaceState(history.state,'',routeUrl(view)); apply(); }));
|
||||
document.querySelector('[data-report-reset]').addEventListener('click', () => { clearFilters(); history.replaceState(history.state,'',routeUrl(view)); apply(); query.focus(); });
|
||||
document.querySelectorAll('[data-report-feature-choice]').forEach(button => button.addEventListener('click', () => { feature.value = button.dataset.reportFeatureChoice; navigate(routeUrl('all','','#results')); document.querySelector('.feature-coverage').open = false; }));
|
||||
document.addEventListener('keydown', event => { if(event.key === '/' && view !== 'case' && !event.metaKey && !event.ctrlKey && !event.altKey && !event.target.matches('input,select,textarea,[contenteditable]')) { event.preventDefault(); query.focus(); } });
|
||||
window.addEventListener('popstate', event => { if(!readRoute())return; restoring = true; apply(); if(typeof event.state?.reportScroll === 'number')window.scrollTo({top:event.state.reportScroll,behavior:'instant'});else if(location.hash)revealTarget(location.hash);else window.scrollTo({top:0,behavior:'instant'});restoring=false; });
|
||||
let frame = 0;
|
||||
window.addEventListener('scroll', () => { if(frame)return;frame=requestAnimationFrame(() => { frame=0;savePosition();const current=sections.find(section=>!section.hidden && section.getBoundingClientRect().top<=130 && section.getBoundingClientRect().bottom>130);document.querySelectorAll('[data-nav-suite-link]').forEach(link=>link.classList.toggle('in-view',view!=='case' && link.dataset.navSuiteLink===current?.dataset.reportSuiteSection)); }); },{passive:true});
|
||||
if(readRoute()) { apply(); if(location.hash)revealTarget(location.hash); }
|
||||
})();
|
||||
`;
|
||||
@@ -0,0 +1,32 @@
|
||||
import type { RunnerE2EBillingSummary } from "./types.js";
|
||||
import { billingCoverageLabel } from "./billing.js";
|
||||
|
||||
export function html(value: unknown) {
|
||||
return String(value ?? "")
|
||||
.replaceAll("&", "&")
|
||||
.replaceAll("<", "<")
|
||||
.replaceAll(">", ">")
|
||||
.replaceAll('"', """)
|
||||
.replaceAll("'", "'");
|
||||
}
|
||||
|
||||
export function durationLabel(durationMs: number) {
|
||||
if (durationMs < 1_000) return `${durationMs}ms`;
|
||||
const seconds = Math.round(durationMs / 1_000);
|
||||
if (seconds < 60) return `${seconds}s`;
|
||||
return `${Math.floor(seconds / 60)}m ${seconds % 60}s`;
|
||||
}
|
||||
|
||||
export function tokenLabel(value: number, llm?: RunnerE2EBillingSummary["llm"]) {
|
||||
const formatted = new Intl.NumberFormat("en-US").format(value);
|
||||
return llm ? billingCoverageLabel(formatted, llm.runsWithTokenUsage, llm.runCount) : formatted;
|
||||
}
|
||||
|
||||
export function llmCostLabel(value: number, llm: RunnerE2EBillingSummary["llm"]) {
|
||||
return billingCoverageLabel(usdLabel(value), llm.runsWithReportedCost, llm.runCount);
|
||||
}
|
||||
|
||||
export function usdLabel(value: number | null) {
|
||||
if (value === null) return "Unknown";
|
||||
return `$${value.toFixed(value < 0.01 ? 6 : 4)}`;
|
||||
}
|
||||
@@ -0,0 +1,311 @@
|
||||
import type { RunnerE2EHistoryIndex } from "./types.js";
|
||||
import { html, durationLabel, tokenLabel, llmCostLabel, usdLabel } from "./report-format.js";
|
||||
|
||||
function renderTrendChart(input: {
|
||||
history: RunnerE2EHistoryIndex;
|
||||
label: string;
|
||||
value(campaign: RunnerE2EHistoryIndex["campaigns"][number]): number | null;
|
||||
format(value: number): string;
|
||||
include?(campaign: RunnerE2EHistoryIndex["campaigns"][number]): boolean;
|
||||
fingerprint?(campaign: RunnerE2EHistoryIndex["campaigns"][number]): string;
|
||||
}) {
|
||||
const campaigns = input.history.campaigns
|
||||
.filter(input.include ?? ((campaign) => campaign.complete))
|
||||
.slice(0, 20)
|
||||
.reverse();
|
||||
if (campaigns.length === 0) {
|
||||
return `<article class="trend-card"><span>${html(input.label)}</span><strong>No complete campaigns</strong></article>`;
|
||||
}
|
||||
const rawValues = campaigns.map(input.value);
|
||||
const values = rawValues.filter((value): value is number => value !== null);
|
||||
if (values.length !== rawValues.length) {
|
||||
const latest = rawValues.at(-1);
|
||||
return `<article class="trend-card"><span>${html(input.label)}</span><strong>${latest == null ? "Unknown" : html(input.format(latest))}</strong><small>Unknown measurements are not plotted. Inspect campaign billing and reserved judge budgets.</small></article>`;
|
||||
}
|
||||
const maximum = Math.max(...values, 1);
|
||||
const pointRows = values.map((value, index) => {
|
||||
const x =
|
||||
campaigns.length === 1 ? 50 : (index / (campaigns.length - 1)) * 100;
|
||||
const y = 96 - (value / maximum) * 88;
|
||||
return {
|
||||
point: `${x.toFixed(2)},${y.toFixed(2)}`,
|
||||
x,
|
||||
y,
|
||||
fingerprint: input.fingerprint?.(campaigns[index]!) ?? "stable",
|
||||
};
|
||||
});
|
||||
const segments = pointRows.reduce<Array<typeof pointRows>>(
|
||||
(groups, point) => {
|
||||
const current = groups.at(-1);
|
||||
if (!current || current.at(-1)?.fingerprint !== point.fingerprint) {
|
||||
groups.push([point]);
|
||||
} else {
|
||||
current.push(point);
|
||||
}
|
||||
return groups;
|
||||
},
|
||||
[],
|
||||
);
|
||||
const definitionCount = new Set(pointRows.map((point) => point.fingerprint))
|
||||
.size;
|
||||
const latest = values.at(-1) ?? 0;
|
||||
return `<article class="trend-card">
|
||||
<span>${html(input.label)}</span>
|
||||
<strong>${html(input.format(latest))}</strong>
|
||||
<svg viewBox="0 0 100 100" preserveAspectRatio="none" role="img" aria-label="${html(input.label)} across ${campaigns.length} complete campaign(s)">
|
||||
${segments.map((segment) => `<polyline points="${segment.map((point) => point.point).join(" ")}" fill="none" vector-effect="non-scaling-stroke" />`).join("")}
|
||||
${pointRows.map((point) => `<circle cx="${point.x.toFixed(2)}" cy="${point.y.toFixed(2)}" r="1.3" vector-effect="non-scaling-stroke" />`).join("")}
|
||||
</svg>
|
||||
<small>${campaigns.length} complete campaign${campaigns.length === 1 ? "" : "s"} · ${definitionCount} definition${definitionCount === 1 ? "" : "s"}</small>
|
||||
</article>`;
|
||||
}
|
||||
|
||||
function suiteSummaryFor(
|
||||
campaign: RunnerE2EHistoryIndex["campaigns"][number],
|
||||
suiteId: string,
|
||||
) {
|
||||
return campaign.suites.find((suite) => suite.suiteId === suiteId);
|
||||
}
|
||||
|
||||
export function renderReportHistory(history: RunnerE2EHistoryIndex | undefined) {
|
||||
if (!history || history.campaigns.length === 0) {
|
||||
return `<section class="history-section" id="history"><div class="section-heading"><div><p class="eyebrow">History</p><h2>Campaign trends</h2></div><p>No historical campaigns have been published yet.</p></div></section>`;
|
||||
}
|
||||
const suiteIds = [
|
||||
...new Set(
|
||||
history.campaigns.flatMap((campaign) =>
|
||||
campaign.suites.map((suite) => suite.suiteId),
|
||||
),
|
||||
),
|
||||
];
|
||||
const charts = [
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Observed + estimated cost",
|
||||
value: (campaign) => campaign.billing.observedAndEstimatedCostUsd,
|
||||
format: usdLabel,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Total tokens",
|
||||
value: (campaign) => campaign.billing.llm.totalTokens,
|
||||
format: tokenLabel,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Agent execution time",
|
||||
value: (campaign) => campaign.billing.agentRunDurationMs,
|
||||
format: durationLabel,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Daytona lease time",
|
||||
value: (campaign) => campaign.billing.leaseDurationMs,
|
||||
format: durationLabel,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
renderTrendChart({
|
||||
history,
|
||||
label: "Pass rate",
|
||||
value: (campaign) =>
|
||||
campaign.selected > 0 ? (campaign.passed / campaign.selected) * 100 : 0,
|
||||
format: (value) => `${value.toFixed(1)}%`,
|
||||
fingerprint: (campaign) =>
|
||||
campaign.suites
|
||||
.map((suite) => `${suite.suiteId}:${suite.suiteDefinitionHash}`)
|
||||
.sort()
|
||||
.join("|"),
|
||||
}),
|
||||
].join("");
|
||||
const suiteCharts = suiteIds
|
||||
.map((suiteId) => {
|
||||
const include = (campaign: RunnerE2EHistoryIndex["campaigns"][number]) =>
|
||||
suiteSummaryFor(campaign, suiteId)?.complete === true;
|
||||
const value = (
|
||||
campaign: RunnerE2EHistoryIndex["campaigns"][number],
|
||||
metric: "cost" | "tokens" | "agent" | "lease" | "passRate",
|
||||
) => {
|
||||
const suite = suiteSummaryFor(campaign, suiteId);
|
||||
if (!suite) return 0;
|
||||
if (metric === "cost") return suite.billing.observedAndEstimatedCostUsd;
|
||||
if (metric === "tokens") return suite.billing.llm.totalTokens;
|
||||
if (metric === "agent") return suite.billing.agentRunDurationMs;
|
||||
if (metric === "lease") return suite.billing.leaseDurationMs;
|
||||
return suite.selected > 0 ? (suite.passed / suite.selected) * 100 : 0;
|
||||
};
|
||||
const fingerprint = (
|
||||
campaign: RunnerE2EHistoryIndex["campaigns"][number],
|
||||
) => suiteSummaryFor(campaign, suiteId)?.suiteDefinitionHash ?? "unknown";
|
||||
return `<section class="suite-trends" data-history-suite-trends="${html(suiteId)}">
|
||||
<div class="suite-trends-heading"><h3>${html(suiteId)}</h3><span>Complete suite selections only; lines break at definition changes.</span></div>
|
||||
<div class="trend-grid">
|
||||
${renderTrendChart({ history, label: "Suite cost", value: (campaign) => value(campaign, "cost"), format: usdLabel, include, fingerprint })}
|
||||
${renderTrendChart({ history, label: "Suite tokens", value: (campaign) => value(campaign, "tokens"), format: tokenLabel, include, fingerprint })}
|
||||
${renderTrendChart({ history, label: "Suite agent time", value: (campaign) => value(campaign, "agent"), format: durationLabel, include, fingerprint })}
|
||||
${renderTrendChart({ history, label: "Suite lease time", value: (campaign) => value(campaign, "lease"), format: durationLabel, include, fingerprint })}
|
||||
${renderTrendChart({ history, label: "Suite pass rate", value: (campaign) => value(campaign, "passRate"), format: (metric) => `${metric.toFixed(1)}%`, include, fingerprint })}
|
||||
</div>
|
||||
</section>`;
|
||||
})
|
||||
.join("");
|
||||
const rows = history.campaigns
|
||||
.map((campaign) => {
|
||||
const status = campaign.failed > 0 ? "failed" : (campaign.incomplete ?? 0) > 0 ? "incomplete" : "passed";
|
||||
const sha = campaign.source.sha;
|
||||
const searchable = [
|
||||
campaign.campaignId,
|
||||
sha,
|
||||
campaign.source.ref,
|
||||
...campaign.executions.flatMap((execution) => [
|
||||
execution.suiteId,
|
||||
execution.profileId,
|
||||
execution.model,
|
||||
execution.environmentId,
|
||||
execution.caseId,
|
||||
execution.status,
|
||||
]),
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(" ")
|
||||
.toLowerCase();
|
||||
return `<tr data-history-campaign data-history-date="${html(campaign.generatedAt.slice(0, 10))}" data-history-status="${status}" data-history-complete="${campaign.complete}" data-history-suites="${html(campaign.suites.map((suite) => suite.suiteId).join(" "))}" data-history-search="${html(searchable)}">
|
||||
<td><a href="${html(campaign.publicUrl)}">${html(campaign.campaignId)}</a><small>${html(new Date(campaign.generatedAt).toLocaleString("en-US", { timeZone: "UTC" }))} UTC</small></td>
|
||||
<td>${sha ? `<code>${html(sha.slice(0, 10))}</code>` : "Unknown"}<small>${html(campaign.source.ref ?? "unknown ref")}</small></td>
|
||||
<td><span class="status history-${status}">${status}</span><small>${campaign.passed}/${campaign.selected} passed${campaign.incomplete ? ` · ${campaign.incomplete} incomplete` : ""} · ${campaign.complete ? "complete" : "partial"}</small></td>
|
||||
<td>${html(tokenLabel(campaign.billing.llm.inputTokens, campaign.billing.llm))} / ${html(tokenLabel(campaign.billing.llm.outputTokens, campaign.billing.llm))}<small>input / output · ${html(tokenLabel(campaign.billing.llm.cachedInputTokens, campaign.billing.llm))} cached</small></td>
|
||||
<td>${html(llmCostLabel(campaign.billing.reportedLlmCostUsd, campaign.billing.llm))}<small>${html(usdLabel(campaign.billing.estimatedRuntimeCostUsd))} runtime estimate</small></td>
|
||||
<td>${html(durationLabel(campaign.billing.agentRunDurationMs))}<small>${html(durationLabel(campaign.billing.leaseDurationMs))} lease</small></td>
|
||||
</tr>`;
|
||||
})
|
||||
.join("");
|
||||
const latest = history.campaigns.find(
|
||||
(campaign) => campaign.campaignId === history.latestCampaignId,
|
||||
);
|
||||
const latestGreen = history.campaigns.find(
|
||||
(campaign) => campaign.campaignId === history.latestGreenCampaignId,
|
||||
);
|
||||
return `<section class="history-section" id="history">
|
||||
<div class="section-heading"><div><p class="eyebrow">History</p><h2>Campaign trends</h2></div><p>Complete campaigns are compared by default. Partial smoke runs remain searchable and are labeled explicitly.</p></div>
|
||||
<nav class="history-pointers" aria-label="Campaign pointers">
|
||||
${latest ? `<a href="${html(latest.publicUrl)}"><span>Latest run</span><strong>${html(latest.campaignId)}</strong></a>` : ""}
|
||||
${latestGreen ? `<a href="${html(latestGreen.publicUrl)}"><span>Latest green</span><strong>${html(latestGreen.campaignId)}</strong></a>` : ""}
|
||||
</nav>
|
||||
<div class="trend-grid">${charts}</div>
|
||||
${suiteCharts}
|
||||
<div class="history-filters">
|
||||
<label>Search <input type="search" data-history-query placeholder="SHA, model, profile, case"></label>
|
||||
<label>Suite <select data-history-suite><option value="">All suites</option>${suiteIds.map((suiteId) => `<option value="${html(suiteId)}">${html(suiteId)}</option>`).join("")}</select></label>
|
||||
<label>Status <select data-history-status><option value="">All statuses</option><option value="passed">Passed</option><option value="failed">Failed</option><option value="incomplete">Incomplete</option></select></label>
|
||||
<label>From <input type="date" data-history-from></label>
|
||||
<label>Through <input type="date" data-history-through></label>
|
||||
<label class="history-checkbox"><input type="checkbox" data-history-partial> Include partial campaigns</label>
|
||||
</div>
|
||||
<div class="history-table-wrap"><table class="history-table"><thead><tr><th>Campaign</th><th>Paperclip SHA</th><th>Result</th><th>Tokens</th><th>Cost</th><th>Execution</th></tr></thead><tbody>${rows}</tbody></table></div>
|
||||
<p class="history-empty" data-history-empty hidden>No campaigns match these filters.</p>
|
||||
</section>`;
|
||||
}
|
||||
|
||||
export const reportHistoryStyles = `
|
||||
.history-pointers { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 12px; margin-bottom: 16px; }
|
||||
.history-pointers a { padding: 14px 16px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--card); text-decoration: none; }
|
||||
.history-pointers span, .history-pointers strong { display: block; }
|
||||
.history-pointers span { color: var(--muted-foreground); font-size: 10px; letter-spacing: .06em; text-transform: uppercase; }
|
||||
.history-pointers strong { margin-top: 4px; font: 550 12px/1.4 var(--font-mono); overflow-wrap: anywhere; }
|
||||
.trend-grid { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 12px; }
|
||||
.trend-card { min-width: 0; padding: 14px 16px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--card); }
|
||||
.trend-card > span, .trend-card > strong, .trend-card > small { display: block; }
|
||||
.trend-card > span, .trend-card > small { color: var(--muted-foreground); font-size: 10px; }
|
||||
.trend-card > span { letter-spacing: .06em; text-transform: uppercase; }
|
||||
.trend-card > strong { margin-top: 4px; font: 550 17px/1.3 var(--font-mono); }
|
||||
.trend-card svg { width: 100%; height: 72px; margin: 10px 0 4px; overflow: visible; }
|
||||
.trend-card polyline { stroke: var(--foreground); stroke-width: 1.6; }
|
||||
.trend-card circle { fill: var(--background); stroke: var(--foreground); stroke-width: 1; }
|
||||
.suite-trends { margin-top: 28px; padding-top: 20px; border-top: 1px solid var(--border); }
|
||||
.suite-trends-heading { display: flex; align-items: baseline; justify-content: space-between; gap: 16px; margin-bottom: 12px; }
|
||||
.suite-trends-heading h3 { margin: 0; font: 600 14px/1.4 var(--font-mono); }
|
||||
.suite-trends-heading span { color: var(--muted-foreground); font-size: 10px; }
|
||||
.history-filters { display: grid; grid-template-columns: minmax(220px, 1fr) repeat(4, minmax(130px, 180px)) auto; align-items: end; gap: 12px; margin: 24px 0 12px; }
|
||||
.history-filters label { display: grid; gap: 5px; color: var(--muted-foreground); font-size: 10px; letter-spacing: .05em; text-transform: uppercase; }
|
||||
.history-filters input, .history-filters select { width: 100%; min-height: 38px; padding: 7px 9px; border: 1px solid var(--border-strong); border-radius: calc(var(--radius) * .8); background: var(--background); color: var(--foreground); }
|
||||
.history-filters .history-checkbox { display: flex; align-items: center; min-height: 38px; padding-bottom: 7px; white-space: nowrap; }
|
||||
.history-checkbox input { width: 16px; min-height: 16px; }
|
||||
.history-table-wrap { overflow-x: auto; border-top: 1px solid var(--border); border-bottom: 1px solid var(--border); }
|
||||
.history-table { width: 100%; min-width: 1020px; border-collapse: collapse; }
|
||||
.history-table th, .history-table td { padding: 12px; border-bottom: 1px solid var(--border); text-align: left; vertical-align: top; }
|
||||
.history-table tr:last-child td { border-bottom: 0; }
|
||||
.history-table th { background: var(--raised); color: var(--muted-foreground); font-size: 10px; letter-spacing: .05em; text-transform: uppercase; }
|
||||
.history-table td small { display: block; margin-top: 3px; color: var(--muted-foreground); font-size: 10px; }
|
||||
.history-table .status { margin-bottom: 2px; }
|
||||
.history-passed { border-color: var(--pass-border); background: var(--pass-bg); color: var(--pass-text); }
|
||||
.history-incomplete { border-color: var(--missing-border); background: var(--missing-bg); color: var(--missing-text); }
|
||||
.history-failed { border-color: var(--fail-border); background: var(--fail-bg); color: var(--fail-text); }
|
||||
.history-empty { padding: 24px 0; color: var(--muted-foreground); text-align: center; }
|
||||
.history-section .section-heading { display:block; margin:24px 0; }
|
||||
.history-section .section-heading h2 { font-size:24px; margin:0; }
|
||||
.history-section .section-heading p { color:var(--muted-foreground); font-size:12px; }
|
||||
.history-section .status { border:1px solid; }
|
||||
[data-index-trends] { margin-top:32px; }
|
||||
[data-index-trends] > summary { cursor:pointer; color:var(--muted-foreground); font-size:12px; }
|
||||
@media(max-width:1100px) { .trend-grid { grid-template-columns:repeat(2,minmax(0,1fr)); }.history-filters { grid-template-columns:repeat(2,minmax(0,1fr)); } }
|
||||
@media(max-width:760px) { .trend-grid,.history-pointers,.history-filters { grid-template-columns:1fr; }.history-table-wrap .history-table { display:table; min-width:1020px; }.history-table tbody { display:table-row-group; }.history-table tr { display:table-row; }.history-table th,.history-table td { display:table-cell; }.history-table td::before { display:none; } }
|
||||
`;
|
||||
|
||||
export const reportHistoryScript = `
|
||||
(() => {
|
||||
const rows = Array.from(document.querySelectorAll("[data-history-campaign]"));
|
||||
if (rows.length === 0) return;
|
||||
const query = document.querySelector("[data-history-query]");
|
||||
const suite = document.querySelector("[data-history-suite]");
|
||||
const status = document.querySelector("[data-history-status]");
|
||||
const from = document.querySelector("[data-history-from]");
|
||||
const through = document.querySelector("[data-history-through]");
|
||||
const partial = document.querySelector("[data-history-partial]");
|
||||
const empty = document.querySelector("[data-history-empty]");
|
||||
const suiteTrends = [...document.querySelectorAll("[data-history-suite-trends]")];
|
||||
const apply = () => {
|
||||
const search = query.value.trim().toLowerCase();
|
||||
let visible = 0;
|
||||
rows.forEach((row) => {
|
||||
const matches =
|
||||
(!search || row.dataset.historySearch.includes(search)) &&
|
||||
(!suite.value || row.dataset.historySuites.split(" ").includes(suite.value)) &&
|
||||
(!status.value || row.dataset.historyStatus === status.value) &&
|
||||
(!from.value || row.dataset.historyDate >= from.value) &&
|
||||
(!through.value || row.dataset.historyDate <= through.value) &&
|
||||
(partial.checked || row.dataset.historyComplete === "true");
|
||||
row.hidden = !matches;
|
||||
if (matches) visible += 1;
|
||||
});
|
||||
suiteTrends.forEach((section) => {
|
||||
section.hidden = Boolean(suite.value) && section.dataset.historySuiteTrends !== suite.value;
|
||||
});
|
||||
empty.hidden = visible !== 0;
|
||||
};
|
||||
query.addEventListener("input", apply);
|
||||
suite.addEventListener("change", apply);
|
||||
status.addEventListener("change", apply);
|
||||
from.addEventListener("change", apply);
|
||||
through.addEventListener("change", apply);
|
||||
partial.addEventListener("change", apply);
|
||||
apply();
|
||||
})();
|
||||
`;
|
||||
@@ -0,0 +1,47 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { renderRunnerE2EDashboard } from "./dashboard.js";
|
||||
import { buildRunnerCampaign, campaignHistoryRecord, emptyRunnerHistory, mergeRunnerHistory } from "./history.js";
|
||||
import { renderRunnerHistoryIndex } from "./history-index.js";
|
||||
import { renderReportShell, reportHistoryHref } from "./report-navigation.js";
|
||||
|
||||
describe("evaluation report navigation", () => {
|
||||
it("never gives global pages a campaign suite tree or campaign selector", () => {
|
||||
const campaign = buildRunnerCampaign({campaignId:"global-fixture",generatedAt:"2026-10-20T12:00:00Z",expected:[runnerMatrix[0]!.id],results:[]});
|
||||
const history = mergeRunnerHistory(null,campaignHistoryRecord(campaign,"https://reports.example.test/runner-e2e"));
|
||||
const page = renderRunnerHistoryIndex(history);
|
||||
const sidebar = page.slice(page.indexOf('<aside '), page.indexOf('</aside>'));
|
||||
expect(sidebar).toContain("Overview");
|
||||
expect(sidebar).toContain("Needs attention");
|
||||
expect(sidebar).toContain("Campaign history");
|
||||
expect(sidebar).not.toContain("Suites");
|
||||
expect(page).not.toContain('aria-label="Campaign suites"');
|
||||
expect(page).not.toContain('data-nav-campaign aria-label="Campaign"');
|
||||
});
|
||||
|
||||
it("lists cases once within selected suites while keeping every execution and missing result", () => {
|
||||
const first = runnerMatrix.find(row => row.suite.id === "everyday-workflows")!;
|
||||
const variants = runnerMatrix.filter(row => row.suite.id === first.suite.id && row.task.id === first.task.id);
|
||||
const page = renderRunnerE2EDashboard({
|
||||
title: "Selected campaign", generatedAt: "2026-10-20T12:00:00Z",
|
||||
expected: variants.map(row => row.id), catalog: runnerMatrix, entries: [],
|
||||
});
|
||||
const sidebar = page.slice(page.indexOf('<aside '), page.indexOf('</aside>'));
|
||||
expect(sidebar.match(/data-nav-case=/g)).toHaveLength(1);
|
||||
expect(sidebar.match(/data-nav-suite=/g)).toHaveLength(1);
|
||||
expect(sidebar).not.toContain('data-nav-suite="core-compatibility"');
|
||||
expect(page.match(/class="case case-missing"/g)).toHaveLength(variants.length);
|
||||
for(const row of variants) expect(page).toContain(`id="execution-${row.id}"`);
|
||||
expect(page).toContain('aria-label="Execution configurations"');
|
||||
});
|
||||
|
||||
it("escapes campaign labels and derives the global history link from the actual report location", () => {
|
||||
const page = renderReportShell({campaignId:'<script>alert("x")</script>', catalog:[]});
|
||||
expect(page).not.toContain('<script>');
|
||||
expect(page).toContain('<script>');
|
||||
const history = emptyRunnerHistory();
|
||||
history.campaigns = [{campaignId:"campaign-a",publicUrl:"/custom/reports/campaigns/campaign-a/"} as typeof history.campaigns[number]];
|
||||
expect(reportHistoryHref(history,"campaign-a")).toBe("/custom/reports/");
|
||||
expect(reportHistoryHref(undefined,"campaign-a")).toBe("../../");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,154 @@
|
||||
import { paperclipReportLogo } from "./report-brand.js";
|
||||
import type { ReportExecution } from "./report-catalog.js";
|
||||
import type { RunnerE2EHistoryIndex } from "./types.js";
|
||||
|
||||
export function reportHtml(value: unknown) {
|
||||
return String(value ?? "").replace(/[&<>"']/g, character =>
|
||||
({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" })[character]!,
|
||||
);
|
||||
}
|
||||
|
||||
export function reportHistoryHref(history: RunnerE2EHistoryIndex | undefined, campaignId: string | undefined) {
|
||||
const url = history?.campaigns.find(campaign => campaign.campaignId === campaignId)?.publicUrl;
|
||||
return url?.replace(/campaigns\/[^/]+\/(?:index\.html)?$/, "") ?? "../../";
|
||||
}
|
||||
|
||||
export function renderReportShell(input: {
|
||||
campaignId?: string;
|
||||
catalog?: readonly ReportExecution[];
|
||||
history?: RunnerE2EHistoryIndex;
|
||||
}) {
|
||||
const h = reportHtml;
|
||||
const campaign = Boolean(input.catalog);
|
||||
const label = input.campaignId ?? "Current campaign";
|
||||
const root = campaign ? reportHistoryHref(input.history, input.campaignId) : "./";
|
||||
const suites = [...new Map((input.catalog ?? []).map(row => [row.suite.id, row.suite])).values()];
|
||||
const tree = campaign ? `<nav class="report-suite-tree" aria-label="Campaign suites"><p class="report-side-label">Suites</p>${suites.map(suite => {
|
||||
const cases = [...new Map(input.catalog!.filter(row => row.suite.id === suite.id).map(row => [row.task.id, row])).values()];
|
||||
return `<details data-nav-suite="${h(suite.id)}"><summary><a href="#suite-${h(suite.id)}" data-nav-suite-link="${h(suite.id)}">${h(suite.label)}<small>${cases.length}</small></a></summary><ul>${cases.map(row => `<li><a href="?view=case&case=${encodeURIComponent(`${suite.id}.${row.task.id}`)}#execution-${h(row.id)}" data-nav-case="${h(`${suite.id}.${row.task.id}`)}" data-report-case-link="${h(`${suite.id}.${row.task.id}`)}"><span class="case-status-dot" aria-hidden="true"></span><span>${h(row.task.label)}</span></a></li>`).join("")}</ul></details>`;
|
||||
}).join("")}</nav>` : "";
|
||||
return `<div class="report-shell" data-report-scope="${campaign ? "campaign" : "global"}" data-campaign-label="${h(label)}">
|
||||
<button class="report-drawer-backdrop" type="button" data-nav-close aria-label="Close navigation" hidden></button>
|
||||
<aside class="report-sidebar" id="report-navigation" aria-label="Report navigation">
|
||||
<a class="brand-lockup report-brand" href="${h(root)}" aria-label="Paperclip evaluations">${paperclipReportLogo}<small>Evaluations</small></a>
|
||||
<div class="report-sidebar-scroll"><nav class="report-global-nav" aria-label="Evaluations">
|
||||
<a href="${campaign ? "?view=all" : "?view=overview"}" data-nav-view="${campaign ? "all" : "overview"}" aria-current="page"><span aria-hidden="true">◫</span>${campaign ? "All results" : "Overview"}</a>
|
||||
<a href="?view=attention" data-nav-view="attention"><span aria-hidden="true">○</span>Needs attention</a>
|
||||
<a href="${h(root)}?view=history" ${campaign ? "" : 'data-nav-view="history"'}><span aria-hidden="true">↶</span>Campaign history</a>
|
||||
</nav>${tree}</div><div class="report-sidebar-footer">Product E2E${campaign ? `<span>${input.catalog!.length} executions</span>` : ""}</div>
|
||||
</aside><div class="report-surface">
|
||||
<header class="report-topbar"><button type="button" class="report-menu-toggle" data-nav-toggle aria-label="Open navigation" aria-controls="report-navigation" aria-expanded="false">☰</button>
|
||||
<nav class="report-breadcrumbs" aria-label="Breadcrumb" data-nav-breadcrumbs><a href="${h(root)}?view=history">Campaigns</a>${campaign ? `<span aria-hidden="true">/</span><span>${h(label)}</span><span aria-hidden="true">/</span><span aria-current="page">All results</span>` : '<span aria-hidden="true">/</span><span aria-current="page">Overview</span>'}</nav>
|
||||
${campaign && input.history?.campaigns.length ? `<select class="report-campaign-select" data-nav-campaign aria-label="Campaign">${!input.history.campaigns.some(item => item.campaignId === input.campaignId) ? `<option selected value="">${h(label)}</option>` : ""}${input.history.campaigns.map(item => `<option value="${h(item.publicUrl)}" ${item.campaignId === input.campaignId ? "selected" : ""}>${h(item.campaignId)}</option>`).join("")}</select>` : ""}
|
||||
</header>`;
|
||||
}
|
||||
|
||||
export const reportShellScript = `
|
||||
(() => {
|
||||
const shell = document.querySelector('.report-shell');
|
||||
const toggle = document.querySelector('[data-nav-toggle]');
|
||||
const backdrop = document.querySelector('[data-nav-close]');
|
||||
const close = () => { shell.classList.remove('navigation-open'); toggle.setAttribute('aria-expanded', 'false'); backdrop.hidden = true; };
|
||||
toggle.addEventListener('click', () => { const open = !shell.classList.contains('navigation-open'); shell.classList.toggle('navigation-open', open); toggle.setAttribute('aria-expanded', String(open)); backdrop.hidden = !open; });
|
||||
backdrop.addEventListener('click', close);
|
||||
shell.querySelectorAll('.report-sidebar a').forEach(link => link.addEventListener('click', close));
|
||||
document.addEventListener('keydown', event => { if(event.key === 'Escape' && shell.classList.contains('navigation-open')) { close(); toggle.focus(); } });
|
||||
document.querySelector('[data-nav-campaign]')?.addEventListener('change', event => { if(event.target.value) location.href = event.target.value; });
|
||||
})();
|
||||
`;
|
||||
|
||||
export const reportNavigationStyles = `
|
||||
:root { --report-sidebar-width: 250px; --navigation-sticky-offset: 64px; }
|
||||
html { scroll-behavior: auto; }
|
||||
[hidden] { display: none !important; }
|
||||
.report-shell { min-height: 100dvh; }
|
||||
.report-surface { margin-left: var(--report-sidebar-width); min-width: 0; }
|
||||
.report-surface main { width: 100%; max-width: 1320px; margin: 0 auto; padding: 32px 36px 72px; }
|
||||
.report-sidebar { position: fixed; inset: 0 auto 0 0; width: var(--report-sidebar-width); display: flex; flex-direction: column; background: var(--card, var(--raised)); border-right: 1px solid var(--border); z-index: 20; }
|
||||
.report-brand { min-height: 68px; padding: 20px; display: flex; align-items: center; gap: 9px; font-size: 14px; text-decoration: none; }
|
||||
.report-brand small { font-size: 10px; font-weight: 400; margin-left: auto; color: var(--muted-foreground); }
|
||||
.report-brand svg { width: auto; height: 24px; flex: none; }
|
||||
.report-sidebar-scroll { min-height: 0; flex: 1; overflow: auto; padding: 0 10px 24px; scrollbar-width: thin; }
|
||||
.report-global-nav { display: grid; gap: 3px; }
|
||||
.report-global-nav a { display: flex; gap: 11px; align-items: center; padding: 9px 10px; font-size: 12px; color: var(--muted-foreground); border-radius: 5px; text-decoration: none; }
|
||||
.report-global-nav a:hover,.report-suite-tree a:hover { background: var(--raised); color: var(--foreground); }
|
||||
.report-global-nav a[aria-current=page] { color: var(--foreground); background: var(--raised); }
|
||||
.report-side-label { padding: 18px 10px 7px; margin: 0; font: 10px var(--font-mono); color: var(--muted-foreground); text-transform: uppercase; letter-spacing: .08em; }
|
||||
.report-suite-tree details > summary { display: flex; align-items: center; gap: 4px; list-style: none; cursor: pointer; padding-left: 4px; }
|
||||
.report-suite-tree summary::-webkit-details-marker { display: none; }
|
||||
.report-suite-tree summary::before { content: '›'; width: 12px; flex: none; color: var(--muted-foreground); }
|
||||
.report-suite-tree details[open] > summary::before { content: '⌄'; }
|
||||
.report-suite-tree a { display: flex; align-items: center; gap: 8px; min-width: 0; flex: 1; padding: 8px 5px; border-radius: 5px; text-decoration: none; font-size: 12px; color: var(--muted-foreground); }
|
||||
.report-suite-tree a small { margin-left: auto; font: 10px var(--font-mono); }
|
||||
.report-suite-tree a[aria-current=page],.report-suite-tree a.in-view { color: var(--foreground); background: var(--raised); }
|
||||
.report-suite-tree ul { display: grid; gap: 2px; margin: 0; padding: 0 0 8px 22px; list-style: none; }
|
||||
.report-suite-tree ul a { font-size: 11px; align-items: baseline; }
|
||||
.report-suite-tree .case-status-dot { flex: 0 0 5px; width: 5px; height: 5px; }
|
||||
.report-sidebar-footer { display: flex; justify-content: space-between; gap: 10px; border-top: 1px solid var(--border); color: var(--muted-foreground); font-size: 10px; padding: 17px 20px; }
|
||||
header.report-topbar { position: sticky; top: 0; z-index: 12; display: flex; align-items: center; justify-content: space-between; flex-wrap: nowrap; min-height: 64px; gap: 20px; margin: 0; padding: 12px 36px; border-bottom: 1px solid var(--border); background: var(--background); }
|
||||
.report-breadcrumbs { display: flex; align-items: center; flex-wrap: wrap; gap: 5px 8px; font-size: 11px; min-width: 0; color: var(--muted-foreground); }
|
||||
.report-breadcrumbs a { text-decoration: none; }
|
||||
.report-breadcrumbs a:hover { text-decoration: underline; }
|
||||
.report-breadcrumbs [aria-current=page] { color: var(--foreground); }
|
||||
.report-breadcrumbs a,.report-breadcrumbs span { overflow-wrap: anywhere; }
|
||||
.report-campaign-select { font: 11px var(--font-sans); padding: 8px; color: var(--foreground); background: var(--card); border: 1px solid var(--border); border-radius: 5px; width: 220px; max-width: 30%; flex: none; }
|
||||
.report-menu-toggle { display: none; border: 0; padding: 0; background: transparent; color: var(--foreground); font-size: 20px; cursor: pointer; }
|
||||
.report-drawer-backdrop { display: none; }
|
||||
.report-header h1 { font-size: 32px; }
|
||||
.report-header .lede { font-size: 12px; }
|
||||
.report-source { display: flex; flex-wrap: wrap; gap: 12px; color: var(--muted-foreground); font-size: 11px; }
|
||||
.report-header { align-items: start; gap: 24px; }
|
||||
.report-actions { flex-wrap: wrap; }
|
||||
.campaign-billing-details { margin: 8px 0 24px; }
|
||||
.campaign-billing-details > summary { cursor: pointer; color: var(--muted-foreground); font-size: 12px; }
|
||||
[data-campaign-overview] .campaign-suites { grid-template-columns: repeat(2,minmax(0,1fr)); gap: 0 28px; }
|
||||
[data-campaign-overview] .campaign-suite-row { grid-template-columns: minmax(0,1fr) auto; gap: 4px 16px; }
|
||||
[data-campaign-overview] .campaign-suite-state { grid-column: 1/-1; text-align: left; font-size: 10px; }
|
||||
.report-surface .report-header { grid-template-columns: 1fr; }
|
||||
.report-surface .results-heading { scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); }
|
||||
.report-surface .suite-section { scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); margin-top: 56px; }
|
||||
.report-surface .section-heading h2 { font-size: 26px; }
|
||||
.report-surface .suite-case-list { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
||||
.report-journey { margin: 32px 0 44px; scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); }
|
||||
.report-journey-heading { display: flex; justify-content: space-between; align-items: baseline; gap: 20px; margin: 0 0 14px; }
|
||||
.report-journey-heading h3 { margin: 0; font-size: 15px; font-weight: 600; }
|
||||
.report-journey-heading a { text-decoration: none; }
|
||||
.report-journey-heading a:hover { text-decoration: underline; }
|
||||
.report-journey-heading > a { font-size: 11px; color: var(--muted-foreground); white-space: nowrap; }
|
||||
.execution-previews { display: grid; grid-template-columns: repeat(auto-fit,minmax(min(100%,390px),1fr)); gap: 20px; align-items: start; }
|
||||
.execution-preview { min-width: 0; }
|
||||
.execution-preview-label { display: flex; align-items: baseline; flex-wrap: wrap; gap: 8px; margin-bottom: 8px; font-size: 10px; color: var(--muted-foreground); }
|
||||
.execution-preview-label strong { color: var(--foreground); font-weight: 500; }
|
||||
.report-surface .profile-sticky { position: static; }
|
||||
.execution-preview .case { scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); }
|
||||
.case-open { text-decoration: none; }
|
||||
.case-open:hover { text-decoration: underline; }
|
||||
.case-config-options { display: flex; flex-wrap: wrap; gap: 7px; margin-bottom: 16px; }
|
||||
.case-config-options button { display: flex; align-items: center; gap: 7px; cursor: pointer; padding: 7px 9px; font-size: 11px; color: var(--muted-foreground); background: var(--background); border: 1px solid var(--border); border-radius: 5px; }
|
||||
.case-config-options button[aria-pressed=true] { color: var(--foreground); background: var(--raised); border-color: var(--border-strong); }
|
||||
.case-config-options button:focus-visible,.report-menu-toggle:focus-visible { outline: 2px solid currentColor; outline-offset: 3px; }
|
||||
.case-page-heading { margin-bottom: 26px; }
|
||||
.case-page-heading h1 { font-size: 28px; margin-bottom: 12px; }
|
||||
.case-page-heading a { font-size: 12px; color: var(--muted-foreground); }
|
||||
.report-shell[data-view=case] .report-journey-heading,.report-shell[data-view=case] .suite-case-toc,.report-shell[data-view=case] .suite-section > .section-heading,.report-shell[data-view=case] .suite-summary,.report-shell[data-view=case] .execution-preview-label { display: none; }
|
||||
.report-shell[data-view=case] .suite-section,.report-shell[data-view=case] .report-journey { margin: 0; }
|
||||
.report-shell[data-view=case] .execution-previews { display: block; }
|
||||
.report-shell[data-view=case] .case { scroll-margin-top: calc(var(--navigation-sticky-offset) + 24px); }
|
||||
@media(max-width:1100px) { :root { --report-sidebar-width: 220px; }.report-surface main { padding: 28px 24px 60px; }header.report-topbar { padding: 12px 24px; }.report-campaign-select { max-width: 35%; }.report-surface .suite-summary { grid-template-columns: repeat(3,minmax(0,1fr)); } }
|
||||
@media(max-width:760px) {
|
||||
.report-surface { margin-left: 0; }.report-surface main { padding: 24px 17px 56px; }
|
||||
.report-sidebar { display: none; width: min(290px,85vw); }.navigation-open .report-sidebar { display: flex; }
|
||||
.navigation-open .report-drawer-backdrop { display: block; position: fixed; inset: 0; border: 0; background: var(--overlay,#0009); z-index: 19; }
|
||||
header.report-topbar { display: grid; grid-template-columns: 24px minmax(0,1fr); gap: 10px; padding: 12px 17px; }
|
||||
.report-menu-toggle { display: block; grid-row: 1; grid-column: 1; }.report-campaign-select { grid-row: 1; grid-column: 2; justify-self: end; max-width: 100%; width: 220px; }
|
||||
.report-breadcrumbs { grid-column: 1/-1; font-size: 10px; }
|
||||
.report-surface .suite-case-list { grid-template-columns: 1fr; }
|
||||
[data-campaign-overview] .campaign-suites { grid-template-columns: 1fr; }
|
||||
.report-surface .suite-summary { grid-template-columns: repeat(2,minmax(0,1fr)); }
|
||||
.report-surface .report-header h1 { font-size: 28px; }.report-surface .section-heading h2 { font-size: 24px; }
|
||||
.execution-previews { grid-template-columns: 1fr; }.report-journey-heading { align-items: start; }.report-journey-heading > a { max-width: 80px; white-space: normal; text-align: right; }
|
||||
.report-surface .report-filters input,.report-surface .report-filters select { font-size: 16px; }
|
||||
.report-shell[data-view=case] .gallery { grid-template-columns: 1fr; }
|
||||
}
|
||||
@media print { .report-sidebar,.report-topbar { display: none; }.report-surface { margin-left: 0; }.report-surface main { padding: 0; } }
|
||||
`;
|
||||
@@ -324,7 +324,7 @@ describe("runner E2E report aggregation", () => {
|
||||
);
|
||||
expect(dashboard).not.toContain('class="case case-not-selected"');
|
||||
expect(dashboard).toContain("<img");
|
||||
expect(dashboard).toContain('class="brand-lockup"');
|
||||
expect(dashboard).toContain('class="brand-lockup report-brand"');
|
||||
expect(dashboard).toContain("data-gallery-dialog");
|
||||
expect(dashboard).toContain(
|
||||
`id="execution-core-compatibility.${executionId}"`,
|
||||
@@ -350,8 +350,8 @@ describe("runner E2E report aggregation", () => {
|
||||
expect(dashboard).toContain("1,250 in · 75 out");
|
||||
expect(dashboard).toContain("$0.0125");
|
||||
expect(dashboard).toContain("unpriced or unavailable runs are excluded");
|
||||
expect(dashboard).toContain('class="profile-sticky"');
|
||||
expect(dashboard).toContain('class="mobile-environment-header"');
|
||||
expect(dashboard).toContain('class="execution-preview-label"');
|
||||
expect(dashboard).toContain('aria-label="Campaign suites"');
|
||||
expect(dashboard).toContain("data-gallery-profile=");
|
||||
expect(dashboard).toContain("data-gallery-environment=");
|
||||
expect(dashboard).toContain("data-gallery-duration=");
|
||||
@@ -366,7 +366,7 @@ describe("runner E2E report aggregation", () => {
|
||||
expect(dashboard).toContain("Feature coverage");
|
||||
expect(dashboard).not.toContain(".report-filters { position: sticky");
|
||||
expect(dashboard).toContain("table-layout: fixed");
|
||||
expect(dashboard).toContain('class="profile-column"');
|
||||
expect(dashboard).toContain('class="execution-previews"');
|
||||
expect(dashboard).toContain('aria-label="Previous"');
|
||||
expect(dashboard).toContain('aria-label="Next"');
|
||||
expect(dashboard).not.toContain("overflow: auto; max-height: calc(100vh");
|
||||
|
||||
Reference in new issue
Block a user