mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 15:35:12 +01:00
Improve runner E2E report navigation and local previews (#16996)
## Thinking Path > - Paperclip is the open source app that people use to manage AI agents for work. > - Product E2E campaigns check user journeys across agent profiles and environments. > - Readers use the campaign history and reports to inspect results and retained evidence. > - The reports showed image summaries, many unselected suites, and a long sticky index. > - Individual case cards also showed routine diagnostics before the visual evidence. > - This pull request adds text summaries, feature filters, folding case indexes, and simpler cards. > - A local preview uses retained public data so designers can change reports without another paid campaign. ## Linked Issues or Issue Description **Current behavior** The report is hard to scan as the journey catalog grows. Suite links can point into hidden results. Case indexes do not show whether a case passed across its configurations. Design changes also need a realistic local preview. **Proposed behavior** Show the selected campaign as text. Link journeys to current feature-map capabilities. Add folding case indexes with status indicators, reliable section links, and a back-to-top control. Keep failed outcomes visible and put routine diagnostics under Test details. Render retained public campaign data with local templates. **Reason and benefit** Readers can find a feature or failed case faster. Designers can iterate against real retained data without starting an application instance or using provider credentials. **Breaking changes** No result schema or grading change. The report stops embedding the generated summary PNG. The publisher still produces that artifact for existing consumers. Related work: #12889 added the existing matrix browser. #12846 concerns the public image publication boundary. This PR changes report presentation and local preview behavior. ## What Changed - Replace image summaries with shared HTML suite summaries on the history and campaign pages. - Show selected executions and preserve missing results for selected cells. - Add explicit journey-to-capability associations, feature filters, and a coverage view with unmapped journeys. - Add folding case indexes, section case lists, and red, green, or amber status indicators across matching configurations. - Fix anchor navigation through filters and add a floating back-to-top control. - Simplify case cards. Keep failure reasons visible. Fold routine checks, billing, and context into Test details. - Add an original-image link to the gallery. - Add a local preview utility with public data caching and offline mode. Document the direct Node command. - Test manifest-listed evidence, rejection of unlisted files and encoded traversal, cache reuse, and offline misses with a real loopback server and a stub downloader. ## Verification - Focused report, history, and preview tests: 75 passed in 6 files. - Report Playwright spec: 4 passed in Chrome with a standalone config that does not start the application server. - `node scripts/check-maps.mjs`: passed. - `git diff --check`: passed. - Manually checked retained campaign reports, feature filters, folding indexes, failure links, card details, gallery links, mobile layout, and keyboard navigation. - `pnpm build`: passed. - `pnpm -r typecheck`: passed. - `pnpm test:e2e:runner:typecheck`: passed. - `pnpm test:run`: stopped after 15 native session database cases failed. The local Postgres package first lacked dynamic-library links. After link hydration, the same cases still failed. A direct `initdb` check then reported `could not create semaphores: No space left on device` and identified the macOS semaphore limit. The complete local suite has not passed. - `pnpm test:e2e:runner:unit`: 3,250 passed, 6 failed, 2 skipped. All six failures reproduced against unchanged `origin/master` report sources with the same installed dependencies. They concern Copilot admission and its Daytona image input, plus the launcher test that runs those assertions. No report or history test failed. - No live provider evaluation was run for this presentation change. - CI on `b1a9c06823` first failed browser shard 3/8. The agent-chat split-handoff test reached cleanup, where cancelling a run returned HTTP 500 from a Postgres deadlock. Cleanup did not restore the experiment setting, and a later test failed its initial setting check. This shard does not contain the report spec. One targeted rerun passed, and that full workflow then passed. This shows recovery; no production fix was made for the deadlock. The later preview-test commit requires its own fresh checks. To review the design, run `node --watch --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/preview.ts`. Open the history address printed by the command, then open a campaign. The README covers offline mode and cache refresh. ## Risks - Feature associations use the current map. They describe linked scenarios, not proof that a whole capability is covered. Historical grades remain unchanged. - New or unfamiliar cases remain unmapped until their assertions are inspected. - The preview needs the public history host on first access. Offline mode requires cached files. It binds to the loopback interface and serves manifest-listed public evidence. - Hiding routine diagnostics adds one disclosure step. Failed or incomplete outcomes remain visible. This is report polish. It does not duplicate planned core work in `ROADMAP.md`. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, code execution, and browser automation. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
4145b5048a
commit
2b16262280
17 files changed
+1071
-109
No files matched your search
@@ -178,6 +178,11 @@ describe documentation scope, not runtime health.
|
||||
|
||||
The map documents user journeys. It does not execute them automatically;
|
||||
generated inventories record documentation scope rather than live test health.
|
||||
Product E2E reports reuse these capability IDs through the
|
||||
[journey associations](../../tests/runner-e2e/journey-features.ts). Their coverage
|
||||
view shows which scenarios in a campaign are linked to each capability and the
|
||||
associated test results. An association or passing scenario does not certify
|
||||
the whole capability; unselected and unmapped journeys remain distinct.
|
||||
|
||||
When maintaining a recipe, compare its entry points and expected results with
|
||||
the current source. Check that linked tests still exercise the stated behavior
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
export interface FeatureCapability {
|
||||
id: string;
|
||||
label: string;
|
||||
description: string;
|
||||
verification?: { scope: string; storybook: string; production: string; sources: string };
|
||||
}
|
||||
export interface FeatureMapIndex {
|
||||
sections: Array<{
|
||||
title: string;
|
||||
features: Array<{ id: string; title: string; description: string; capabilities: FeatureCapability[] }>;
|
||||
}>;
|
||||
}
|
||||
export function featureIndex(root: string): FeatureMapIndex;
|
||||
export function sidebarLabel(title: string): string;
|
||||
export function mapIndexes(index: FeatureMapIndex): Map<string, string>;
|
||||
export function pageInventory(root: string, index: FeatureMapIndex): { pages: Array<{ file: string; features: string[] }> };
|
||||
@@ -1,10 +1,11 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
|
||||
import { runnerExecutionById } from "../runner-e2e/catalog.js";
|
||||
import { runnerExecutionById, runnerMatrix } from "../runner-e2e/catalog.js";
|
||||
import {
|
||||
renderRunnerE2EDashboard,
|
||||
type RunnerDashboardEntry,
|
||||
} from "../runner-e2e/dashboard.js";
|
||||
import { buildRunnerCampaign } from "../runner-e2e/history.js";
|
||||
import type { MatrixExecution, RunnerE2EResult } from "../runner-e2e/types.js";
|
||||
|
||||
const executionIds = [
|
||||
@@ -77,7 +78,7 @@ test("filters matrix results and pages through the filtered screenshot gallery",
|
||||
title: "Runner E2E interaction fixture",
|
||||
generatedAt: "2026-09-05T12:01:00.000Z",
|
||||
expected: executionIds,
|
||||
catalog,
|
||||
catalog: runnerMatrix,
|
||||
entries,
|
||||
}),
|
||||
);
|
||||
@@ -86,19 +87,36 @@ test("filters matrix results and pages through the filtered screenshot gallery",
|
||||
.locator("[data-report-query]")
|
||||
.locator("..")
|
||||
.locator("..");
|
||||
const tabs = page.locator(".suite-nav");
|
||||
await expect(page.locator(".suite-nav")).toHaveCount(0);
|
||||
await expect(page.locator(".suite-section")).toHaveCount(2);
|
||||
await expect(page.locator('select[data-report-suite] option')).toHaveCount(3);
|
||||
await expect(page.locator('select[data-report-profile] option')).toHaveCount(4);
|
||||
await expect(page.locator("[data-report-result]")).toHaveText(
|
||||
"4 of 4 tests shown",
|
||||
);
|
||||
await expect(page.locator("[data-gallery-item]")).toHaveCount(4);
|
||||
const passedCard = page.locator('[data-execution-id="core-compatibility.legacy-codex.local.message-marker"]');
|
||||
const failedCard = page.locator('[data-execution-id="core-compatibility.legacy-codex.daytona.message-marker"]');
|
||||
await expect(passedCard.locator(".status")).toHaveText("passed");
|
||||
await expect(passedCard.locator(".case-outcome")).toBeHidden();
|
||||
await expect(passedCard.locator(".billing-strip")).toBeHidden();
|
||||
await expect(passedCard.locator(".execution-id")).toBeHidden();
|
||||
await expect(passedCard.locator(".case-diagnostics > summary")).toContainText("1/1 checks · 12s");
|
||||
await expect(failedCard.locator(":scope > .case-outcome")).toBeVisible();
|
||||
expect(
|
||||
await filters.evaluate((element) => getComputedStyle(element).position),
|
||||
).toBe("static");
|
||||
expect(
|
||||
await tabs.evaluate((element) =>
|
||||
element.nextElementSibling?.classList.contains("report-filters"),
|
||||
),
|
||||
).toBe(true);
|
||||
await page.locator('.feature-coverage > summary').click();
|
||||
await page.locator('.feature-coverage').getByRole('button', { name: 'Questions and approvals', exact: true }).click();
|
||||
await expect(page.locator('select[data-report-feature]')).toHaveValue('questions-and-approvals');
|
||||
await expect(page.locator('.feature-coverage')).not.toHaveAttribute('open');
|
||||
await expect(page.locator('[data-report-result]')).toHaveText('1 of 4 tests shown');
|
||||
await expect(page.locator('.suite-section:visible')).toHaveCount(1);
|
||||
await expect(page.locator('[data-gallery-open]')).toHaveText('View gallery · 1');
|
||||
await page.setViewportSize({ width: 900, height: 900 });
|
||||
await expect(page.locator('[data-report-profile-row]:visible')).toHaveCount(1);
|
||||
await page.setViewportSize({ width: 1280, height: 720 });
|
||||
await page.locator('[data-report-reset]').click();
|
||||
|
||||
const firstTable = page.locator(".matrix").first();
|
||||
const widthsBefore = await firstTable
|
||||
@@ -113,6 +131,7 @@ test("filters matrix results and pages through the filtered screenshot gallery",
|
||||
cells.map((cell) => cell.getBoundingClientRect().width),
|
||||
);
|
||||
expect(widthsAfter).toEqual(widthsBefore);
|
||||
await expect(passedCard.locator(".billing-strip")).toBeVisible();
|
||||
|
||||
await page.locator("[data-report-query]").fill("Legacy Claude");
|
||||
await expect(page.locator("[data-report-result]")).toHaveText(
|
||||
@@ -146,6 +165,7 @@ test("filters matrix results and pages through the filtered screenshot gallery",
|
||||
await expect(dialog.locator("[data-gallery-position]")).toHaveText(
|
||||
"Image 1 of 2",
|
||||
);
|
||||
await expect(dialog.locator("[data-gallery-original]")).toHaveAttribute("href", /legacy-codex\.local\.message-marker\/attempt-1\/final-state\.png$/);
|
||||
|
||||
await page.keyboard.press("ArrowRight");
|
||||
await expect(dialog.locator("[data-gallery-environment]")).toHaveText(
|
||||
@@ -155,6 +175,7 @@ test("filters matrix results and pages through the filtered screenshot gallery",
|
||||
await expect(dialog.locator("[data-gallery-position]")).toHaveText(
|
||||
"Image 2 of 2",
|
||||
);
|
||||
await expect(dialog.locator("[data-gallery-original]")).toHaveAttribute("href", /legacy-codex\.daytona\.message-marker\/attempt-1\/final-state\.png$/);
|
||||
|
||||
await page.keyboard.press("Escape");
|
||||
await expect(dialog).not.toBeVisible();
|
||||
@@ -162,3 +183,158 @@ test("filters matrix results and pages through the filtered screenshot gallery",
|
||||
await page.keyboard.press("/");
|
||||
await expect(page.locator("[data-report-query]")).toBeFocused();
|
||||
});
|
||||
|
||||
function navigationReport(ids: readonly string[], suppliedEntries?: readonly RunnerDashboardEntry[]) {
|
||||
const catalog = ids.map(runnerExecutionById);
|
||||
const entries = suppliedEntries ?? catalog.map((execution) => resultFor(execution, "passed"));
|
||||
const generatedAt = "2026-09-05T12:01:00.000Z";
|
||||
return renderRunnerE2EDashboard({
|
||||
title: "Runner E2E navigation fixture",
|
||||
generatedAt,
|
||||
expected: ids,
|
||||
catalog: runnerMatrix,
|
||||
entries,
|
||||
campaign: buildRunnerCampaign({
|
||||
campaignId: "navigation-fixture",
|
||||
generatedAt,
|
||||
expected: ids,
|
||||
catalog,
|
||||
results: entries.map((entry) => entry.result),
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
test("suite links reveal filtered sections and scroll on repeated clicks", async ({ page }) => {
|
||||
const report = navigationReport(executionIds);
|
||||
const reportUrl = "http://runner-e2e.test/campaign/";
|
||||
await page.route(reportUrl, (route) => route.fulfill({ contentType: "text/html", body: report }));
|
||||
await page.goto(reportUrl);
|
||||
|
||||
const core = page.locator("#suite-core-compatibility");
|
||||
const coreHeading = core.getByRole("heading", { name: "Core Runner Compatibility", exact: true });
|
||||
const coreLink = page.getByRole("link", { name: "Core Runner Compatibility", exact: true });
|
||||
const backToTop = page.getByRole("link", { name: "Back to top", exact: true });
|
||||
await expect(backToTop).toBeHidden();
|
||||
const feature = page.locator("select[data-report-feature]");
|
||||
for (let click = 0; click < 2; click += 1) {
|
||||
await feature.selectOption("questions-and-approvals");
|
||||
await expect(core).toBeHidden();
|
||||
await coreLink.click();
|
||||
await expect(core).toBeVisible();
|
||||
await expect(coreHeading).toBeInViewport();
|
||||
await expect(feature).toHaveValue("");
|
||||
await expect(page).toHaveURL(`${reportUrl}#suite-core-compatibility`);
|
||||
await expect(page.locator("[data-report-result]")).toHaveText("4 of 4 tests shown");
|
||||
}
|
||||
|
||||
await page.locator("select[data-report-profile]").selectOption("legacy-codex");
|
||||
await coreLink.click();
|
||||
await expect(coreHeading).toBeInViewport();
|
||||
await expect(page.locator("select[data-report-profile]")).toHaveValue("legacy-codex");
|
||||
await expect(backToTop).toBeVisible();
|
||||
expect(await backToTop.evaluate((element) => getComputedStyle(element).position)).toBe("fixed");
|
||||
await backToTop.focus();
|
||||
await page.keyboard.press("Enter");
|
||||
await expect.poll(() => page.evaluate(() => window.scrollY)).toBe(0);
|
||||
await expect(backToTop).toBeHidden();
|
||||
await expect(page.locator(".report-header h1")).toBeFocused();
|
||||
await expect(page.locator("select[data-report-profile]")).toHaveValue("legacy-codex");
|
||||
await coreLink.click();
|
||||
|
||||
await page.getByRole("link", { name: "Local Session Integrity", exact: true }).click();
|
||||
await expect(page.locator("#suite-local-session-integrity")).toBeVisible();
|
||||
await feature.selectOption("questions-and-approvals");
|
||||
await page.goBack();
|
||||
await expect(coreHeading).toBeInViewport();
|
||||
await expect(feature).toHaveValue("");
|
||||
});
|
||||
|
||||
test("folds the suite index and links each case to a matching configuration", async ({ page }) => {
|
||||
const ids = [...executionIds, "core-compatibility.legacy-codex.local.ask-question"];
|
||||
const reportUrl = "http://runner-e2e.test/campaign/";
|
||||
await page.route(reportUrl, (route) => route.fulfill({ contentType: "text/html", body: navigationReport(ids) }));
|
||||
await page.goto(reportUrl);
|
||||
const coreIndex = page.locator(".campaign-suite-details").filter({
|
||||
has: page.getByRole("link", { name: "Core Runner Compatibility", exact: true }),
|
||||
});
|
||||
await expect(coreIndex).not.toHaveAttribute("open");
|
||||
await coreIndex.getByRole("link", { name: "Core Runner Compatibility", exact: true }).click();
|
||||
await expect(coreIndex).not.toHaveAttribute("open");
|
||||
await coreIndex.locator("summary").focus();
|
||||
await page.keyboard.press("Enter");
|
||||
await expect(coreIndex).toHaveAttribute("open");
|
||||
await expect(coreIndex.locator(".suite-case-list a")).toHaveCount(2);
|
||||
const sectionToc = page.locator("#suite-core-compatibility .suite-case-toc");
|
||||
await expect(sectionToc.locator(".suite-case-list a")).toHaveCount(2);
|
||||
|
||||
const basicLink = coreIndex.getByRole("link", { name: "Basic response", exact: true });
|
||||
const profile = page.locator("select[data-report-profile]");
|
||||
const feature = page.locator("select[data-report-feature]");
|
||||
await profile.selectOption("legacy-claude");
|
||||
const claudeResult = page.locator('[data-execution-id="core-compatibility.legacy-claude.local.message-marker"]');
|
||||
await basicLink.click();
|
||||
await expect(claudeResult.locator(".case-heading")).toBeInViewport();
|
||||
await expect(profile).toHaveValue("legacy-claude");
|
||||
await expect(page).toHaveURL(`${reportUrl}#execution-core-compatibility.legacy-claude.local.message-marker`);
|
||||
|
||||
for (let click = 0; click < 2; click += 1) {
|
||||
await feature.selectOption("questions-and-approvals");
|
||||
await basicLink.click();
|
||||
await expect(feature).toHaveValue("");
|
||||
await expect(profile).toHaveValue("");
|
||||
await expect(page.locator('[data-execution-id="core-compatibility.legacy-codex.local.message-marker"] .case-heading')).toBeInViewport();
|
||||
}
|
||||
await sectionToc.getByRole("link", { name: "Ask mode question", exact: true }).click();
|
||||
await expect(page.locator('[data-execution-id="core-compatibility.legacy-codex.local.ask-question"] .case-heading')).toBeInViewport();
|
||||
await sectionToc.getByRole("link", { name: "Back to index", exact: false }).click();
|
||||
await expect(coreIndex.locator("summary")).toBeInViewport();
|
||||
|
||||
await page.setViewportSize({ width: 390, height: 844 });
|
||||
await expect(basicLink).toBeVisible();
|
||||
expect(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth)).toBe(true);
|
||||
await basicLink.click();
|
||||
const backToTop = page.getByRole("link", { name: "Back to top", exact: true });
|
||||
await expect(backToTop).toBeVisible();
|
||||
expect(await backToTop.evaluate((element) => {
|
||||
const bounds = element.getBoundingClientRect();
|
||||
return bounds.left >= 0 && bounds.right <= innerWidth && bounds.bottom <= innerHeight;
|
||||
})).toBe(true);
|
||||
await backToTop.click();
|
||||
await expect.poll(() => page.evaluate(() => window.scrollY)).toBe(0);
|
||||
});
|
||||
|
||||
test("case status dots summarize matching results and open failures", async ({ page }) => {
|
||||
const ids = [...executionIds, "core-compatibility.legacy-codex.local.ask-question"];
|
||||
const entries = executionIds.map((id, index) => resultFor(runnerExecutionById(id), index === 1 ? "failed" : "passed"));
|
||||
const reportUrl = "http://runner-e2e.test/campaign/";
|
||||
await page.route(reportUrl, (route) => route.fulfill({ contentType: "text/html", body: navigationReport(ids, entries) }));
|
||||
await page.goto(reportUrl);
|
||||
const toc = page.locator("#suite-core-compatibility .suite-case-toc");
|
||||
const basic = toc.getByRole("link", { name: "Basic response", exact: true });
|
||||
const missing = toc.getByRole("link", { name: "Ask mode question", exact: true });
|
||||
const topLink = page.locator('.campaign-case-index [data-report-case-link="core-compatibility.message-marker"]');
|
||||
await expect(basic).toHaveAttribute("data-case-status", "failed");
|
||||
await expect(topLink).toHaveAttribute("data-case-status", "failed");
|
||||
await expect(basic).toHaveAccessibleDescription("Failed · 2/3 passed · 1 failed — All configurations");
|
||||
await expect(missing).toHaveAttribute("data-case-status", "missing");
|
||||
await expect(missing).toHaveAttribute("title", "Missing result · 0/1 passed · 1 missing — All configurations");
|
||||
const red = await basic.locator(".case-status-dot").evaluate((element) => getComputedStyle(element).backgroundColor);
|
||||
|
||||
const profile = page.locator("select[data-report-profile]");
|
||||
await profile.selectOption("legacy-claude");
|
||||
await expect(basic).toHaveAttribute("data-case-status", "passed");
|
||||
await expect(topLink).toHaveAttribute("data-case-status", "passed");
|
||||
await expect(basic).toHaveAccessibleDescription("Passed · 1/1 passed — Matching configurations");
|
||||
const green = await basic.locator(".case-status-dot").evaluate((element) => getComputedStyle(element).backgroundColor);
|
||||
expect(green).not.toBe(red);
|
||||
await expect(missing).toHaveAttribute("title", /none match current filters/);
|
||||
|
||||
await profile.selectOption("legacy-codex");
|
||||
await expect(basic).toHaveAttribute("data-case-status", "failed");
|
||||
await page.locator("select[data-report-environment]").selectOption("local");
|
||||
await expect(basic).toHaveAttribute("data-case-status", "passed");
|
||||
await page.locator("[data-report-reset]").click();
|
||||
await basic.click();
|
||||
await expect(page).toHaveURL(`${reportUrl}#execution-core-compatibility.legacy-codex.daytona.message-marker`);
|
||||
await expect(page.locator('[data-execution-id="core-compatibility.legacy-codex.daytona.message-marker"] .case-heading')).toBeInViewport();
|
||||
});
|
||||
@@ -253,6 +253,28 @@ pnpm test:e2e:runner -- --list --suite completion-updates
|
||||
pnpm test:e2e:runner -- --suite completion-updates
|
||||
```
|
||||
|
||||
## Local report design preview
|
||||
|
||||
Iterate on the public history and individual campaign reports with retained real
|
||||
data, without starting Paperclip, loading credentials, or calling providers:
|
||||
|
||||
```sh
|
||||
node --watch --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/preview.ts
|
||||
```
|
||||
|
||||
Open `http://127.0.0.1:4317/runner-e2e/`. The preview uses the production
|
||||
`history-index.ts` and `dashboard.ts` renderers. Changes restart the preview
|
||||
and reload the browser. Every history link opens a locally rendered report;
|
||||
normalized results, public screenshots, and other manifest-listed public evidence
|
||||
are downloaded on first access and cached under the ignored
|
||||
`tests/runner-e2e/results/design-preview/` directory. Original grades, billing,
|
||||
timestamps, and source provenance are retained. No results are republished.
|
||||
|
||||
Use `--port 4320` for another port, `--refresh` to refresh the history snapshot,
|
||||
or `--offline` to require cached data. Offline mode supports reports and assets
|
||||
already visited; it reports uncached artifacts explicitly. The preview contains
|
||||
only published evidence, not private Playwright traces or videos.
|
||||
|
||||
## Credentials
|
||||
|
||||
Copy `.env.runner-e2e.example` to `.env.runner-e2e.local` and fill only the
|
||||
@@ -757,7 +779,41 @@ captures remain failed evidence; qualifying a fix requires a new live attempt.
|
||||
Every completed local campaign also writes
|
||||
`tests/runner-e2e/results/<campaign>/dashboard.html`. The self-contained page
|
||||
shows the complete profile/environment grid with screenshot thumbnails.
|
||||
Expanding a case shows its matchers, pass/fail details, provider/model/runtime,
|
||||
The history homepage and campaign report render their campaign summaries as
|
||||
selectable HTML text, with suite links, pass counts, status, and timing.
|
||||
Campaign reports render only selected executions, including selected cases with
|
||||
missing evidence or startup failures. Search and feature/suite/profile filters
|
||||
replace the full-catalog sticky index.
|
||||
Each suite in the campaign report summary folds open to a list of its selected cases.
|
||||
The same case links appear at the start of the suite, with a return to the top
|
||||
index. Cases appear once across agent/environment configurations; links prefer
|
||||
a result matching the active filters and clear conflicting filters when needed
|
||||
to reveal their destination.
|
||||
Case links show a green status dot when all matching configurations passed,
|
||||
red when any failed, and amber for incomplete or missing results. Their tooltip
|
||||
summarizes the configurations behind the status; red links prefer a failing
|
||||
result. The status follows the active filters, falling back to the whole case
|
||||
when none of its configurations match.
|
||||
|
||||
`journey-features.ts` associates exact suite/case identities with the stable
|
||||
capability IDs from `maps/feature-map/`. The feature selector uses the map's
|
||||
existing sections and titles. The expandable Feature coverage view separates
|
||||
capability associations, linked test outcomes, gaps, and unmapped journeys.
|
||||
These display-only associations do not change prompts, graders, definitions,
|
||||
or retained results. Historical reports use the current associations, not a
|
||||
claim that those annotations were recorded by the original run. A linked,
|
||||
passing scenario does not establish complete capability coverage. Extend the
|
||||
associations only after inspecting the journey's assertions; unknown cases stay
|
||||
unmapped rather than inheriting coverage from a similarly named suite.
|
||||
|
||||
Passing cards emphasize screenshots and show checks and duration beside a
|
||||
single Test details disclosure. That disclosure holds billing, evidence links,
|
||||
execution identity, and diagnostics; failed and incomplete results keep their
|
||||
explanations visible. Screenshots open the gallery, which also links to the
|
||||
original image. A floating Back to top control returns to the report heading
|
||||
without clearing filters.
|
||||
|
||||
Expanding Test details shows matchers, pass/fail details, provider/model/runtime,
|
||||
timings, token and cost accounting, and evidence links. The campaign header
|
||||
aggregates input, output, and cached tokens, provider-reported LLM spend,
|
||||
Daytona list-price runtime estimates, and pricing coverage. Missing provider
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
import type { ReportExecution } from "./report-catalog.js";
|
||||
import type { RunnerE2ESuiteSummary } from "./types.js";
|
||||
|
||||
function html(value: unknown) {
|
||||
return String(value ?? "").replace(/[&<>"']/g, (character) =>
|
||||
({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" })[character]!,
|
||||
);
|
||||
}
|
||||
|
||||
function uniqueCases(catalog: readonly ReportExecution[]) {
|
||||
const cases = new Map<string, ReportExecution>();
|
||||
for (const execution of catalog) {
|
||||
const key = `${execution.suite.id}.${execution.task.id}`;
|
||||
if (!cases.has(key)) cases.set(key, execution);
|
||||
}
|
||||
return [...cases.values()];
|
||||
}
|
||||
|
||||
export function renderSuiteCaseIndex(catalog: readonly ReportExecution[]) {
|
||||
return `<ul class="suite-case-list">${uniqueCases(catalog).map((execution) => `<li>
|
||||
<a href="#execution-${html(execution.id)}" data-report-case-link="${html(`${execution.suite.id}.${execution.task.id}`)}"><span class="case-status-dot" aria-hidden="true"></span><span>${html(execution.task.label)}</span></a>
|
||||
</li>`).join("")}</ul>`;
|
||||
}
|
||||
|
||||
export function renderCampaignSuiteSummary(
|
||||
suites: readonly RunnerE2ESuiteSummary[],
|
||||
catalog: readonly ReportExecution[],
|
||||
reportHref = "",
|
||||
showCases = false,
|
||||
) {
|
||||
const labels = new Map(catalog.map((execution) => [execution.suite.id, execution.suite.label]));
|
||||
const selected = suites.filter((suite) => suite.selected > 0);
|
||||
if (selected.length === 0) return "";
|
||||
return `<ul class="campaign-suites" aria-label="Suite results">${selected.map((suite) => {
|
||||
const passed = suite.passed === suite.selected && suite.cleanupPassed;
|
||||
const label = labels.get(suite.suiteId) ?? suite.suiteId;
|
||||
const suiteCatalog = catalog.filter((execution) => execution.suite.id === suite.suiteId);
|
||||
const caseCount = uniqueCases(suiteCatalog).length;
|
||||
const fold = showCases && caseCount > 0;
|
||||
const heading = `<span class="campaign-suite-title"><a href="${html(reportHref)}#suite-${html(suite.suiteId)}">${html(label)}</a>${fold ? `<small>${caseCount} ${caseCount === 1 ? "case" : "cases"}</small>` : ""}</span>
|
||||
<strong>${suite.passed}/${suite.selected}</strong>
|
||||
<span class="campaign-suite-state ${passed ? "campaign-suite-passed" : "campaign-suite-attention"}">${passed ? "Passed" : "Needs attention"}${suite.incomplete ? ` · ${suite.incomplete} incomplete` : ""}</span>`;
|
||||
return `<li>${fold ? `<details class="campaign-suite-details">
|
||||
<summary class="campaign-suite-row">${heading}</summary>
|
||||
<nav class="campaign-case-index" aria-label="${html(label)} cases">${renderSuiteCaseIndex(suiteCatalog)}</nav>
|
||||
</details>` : `<div class="campaign-suite-row">${heading}</div>`}
|
||||
</li>`;
|
||||
}).join("")}</ul>`;
|
||||
}
|
||||
|
||||
export const campaignSuiteSummaryStyles = `
|
||||
.campaign-suites { display: grid; margin: 0 0 32px; padding: 0; list-style: none; }
|
||||
.campaign-suites > li { border-bottom: 1px solid var(--border); }
|
||||
.campaign-suite-row { display: grid; grid-template-columns: minmax(0, 1fr) auto 150px; align-items: baseline; gap: 24px; padding: 12px 0; }
|
||||
.campaign-suite-details > summary { grid-template-columns: 16px minmax(0, 1fr) auto 150px; gap: 12px; cursor: pointer; list-style: none; }
|
||||
.campaign-suite-details > summary::-webkit-details-marker { display: none; }
|
||||
.campaign-suite-details > summary::before { content: "›"; align-self: center; color: var(--muted-foreground, var(--muted)); font-size: 20px; line-height: 1; text-align: center; }
|
||||
.campaign-suite-details[open] > summary::before { transform: rotate(90deg); }
|
||||
.campaign-suite-title { display: flex; flex-wrap: wrap; align-items: baseline; gap: 4px 12px; min-width: 0; }
|
||||
.campaign-suite-title small { color: var(--muted-foreground, var(--muted)); font-size: 11px; }
|
||||
.campaign-case-index { padding: 0 0 16px 28px; }
|
||||
.campaign-suites a { width: fit-content; font-weight: 500; text-decoration: none; }
|
||||
.campaign-suites a:hover { text-decoration: underline; }
|
||||
.campaign-suites strong { font-family: var(--font-mono, var(--mono)); font-variant-numeric: tabular-nums; font-weight: 600; }
|
||||
.campaign-suite-state { font-size: 12px; text-align: right; }
|
||||
.campaign-suite-passed { color: var(--pass-text, var(--pass)); }
|
||||
.campaign-suite-attention { color: var(--fail-text, var(--fail)); }
|
||||
.suite-case-list { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 6px 24px; margin: 0; padding: 0; list-style: none; }
|
||||
.suite-case-list a { display: inline-flex; align-items: baseline; gap: 8px; padding: 4px 0; color: var(--muted-foreground, var(--muted)); font-size: 12px; font-weight: 400; text-decoration: none; text-underline-offset: 3px; overflow-wrap: anywhere; }
|
||||
.suite-case-list a:hover { color: var(--foreground); text-decoration: underline; }
|
||||
.case-status-dot { display: inline-block; flex: 0 0 7px; width: 7px; height: 7px; border-radius: 50%; background: var(--muted-foreground, var(--muted)); }
|
||||
[data-case-status="passed"] .case-status-dot { background: var(--pass-text, var(--pass)); }
|
||||
[data-case-status="failed"] .case-status-dot { background: var(--fail-text, var(--fail)); }
|
||||
[data-case-status="incomplete"] .case-status-dot, [data-case-status="missing"] .case-status-dot { background: var(--missing-text); }
|
||||
@media (max-width: 980px) {
|
||||
.suite-case-list { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
||||
}
|
||||
@media (max-width: 520px) {
|
||||
.campaign-suite-row { grid-template-columns: minmax(0, 1fr) auto; gap: 4px 16px; }
|
||||
.campaign-suite-state { grid-column: 1 / -1; text-align: left; }
|
||||
.campaign-suite-details > summary { grid-template-columns: 16px minmax(0, 1fr) auto; gap: 4px 12px; }
|
||||
.campaign-suite-details .campaign-suite-state { grid-column: 2 / -1; }
|
||||
.suite-case-list { grid-template-columns: 1fr; gap: 2px; }
|
||||
}
|
||||
`;
|
||||
@@ -57,7 +57,6 @@ export async function regenerateRunnerDashboard(input: {
|
||||
historyFile?: string | null;
|
||||
outputDirectory?: string;
|
||||
evidenceHrefPrefix?: string;
|
||||
publicSummaryImageHref?: string;
|
||||
}) {
|
||||
const bundle = path.resolve(input.bundle);
|
||||
const outputDirectory = path.resolve(input.outputDirectory ?? bundle);
|
||||
@@ -158,7 +157,6 @@ export async function regenerateRunnerDashboard(input: {
|
||||
entries,
|
||||
campaign,
|
||||
history,
|
||||
publicSummaryImageHref: input.publicSummaryImageHref,
|
||||
});
|
||||
const upgraded = {
|
||||
...campaign,
|
||||
|
||||
+150
-52
@@ -1,4 +1,7 @@
|
||||
import { isIncompleteFirstTaskResult, renderCaseOutcome } from "./case-outcome.js";
|
||||
import { campaignSuiteSummaryStyles, renderCampaignSuiteSummary, renderSuiteCaseIndex } from "./campaign-summary.js";
|
||||
import { journeyCapabilities, journeyFeatureIds } from "./journey-features.js";
|
||||
import { journeyFeatureStyles, renderJourneyFeatureCoverage, renderJourneyFeatureOptions } from "./feature-coverage.js";
|
||||
import { STARTUP_REUSE_LAYERS, summarizeStartupReuse } from "../../packages/adapter-utils/src/startup-reuse.js";
|
||||
import { renderFirstTaskDetails } from "./first-task-report.js";
|
||||
import {
|
||||
@@ -34,7 +37,6 @@ export interface RunnerDashboardInput {
|
||||
entries: readonly RunnerDashboardEntry[];
|
||||
campaign?: RunnerE2ECampaign;
|
||||
history?: RunnerE2EHistoryIndex;
|
||||
publicSummaryImageHref?: string;
|
||||
}
|
||||
|
||||
interface ResolvedScreenshot {
|
||||
@@ -89,17 +91,6 @@ function safeEvidenceHref(base: string | undefined, relative: string) {
|
||||
return `${cleanBase}/${cleanRelative}`;
|
||||
}
|
||||
|
||||
function safePublicAssetHref(relative: string | undefined) {
|
||||
if (!relative || /^(?:[a-z]+:|\/\/|\/)/i.test(relative)) return null;
|
||||
const segments = relative.split("/");
|
||||
if (
|
||||
segments.some((segment) => !segment || segment === "." || segment === "..")
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
return segments.map(encodeURIComponent).join("/");
|
||||
}
|
||||
|
||||
function compactJson(value: unknown) {
|
||||
const serialized = JSON.stringify(value);
|
||||
return serialized && serialized.length > 1_200
|
||||
@@ -178,6 +169,10 @@ function renderCase(
|
||||
: item,
|
||||
);
|
||||
const billing = entry ? summarizeExecutionBilling(entry.result) : null;
|
||||
const capabilities = journeyCapabilities(execution);
|
||||
const features = [...new Map(capabilities.map((capability) => [capability.feature.id, capability.feature])).values()];
|
||||
const outcome = entry ? renderCaseOutcome(entry.result, entry.valid, entry.errors) : "";
|
||||
const passed = state === "passed" && entry?.result.status === "passed";
|
||||
const firstTaskChecks = entry?.result.firstTask?.checks;
|
||||
const notReached = firstTaskChecks?.filter((c) => c.notReached).length ?? 0;
|
||||
const matcherResults = firstTaskChecks?.length
|
||||
@@ -200,6 +195,7 @@ function renderCase(
|
||||
execution.environment.label,
|
||||
execution.environment.provider,
|
||||
execution.suite.label,
|
||||
...capabilities.flatMap((capability) => [capability.id, capability.label, capability.feature.title]),
|
||||
label,
|
||||
]
|
||||
.join(" ")
|
||||
@@ -222,9 +218,8 @@ function renderCase(
|
||||
return href ? [`<a href="${html(href)}">${html(label)}</a>`] : [];
|
||||
});
|
||||
const links =
|
||||
screenshots.length > 0 || playwright || structuredLinks.length > 0
|
||||
playwright || structuredLinks.length > 0
|
||||
? `<nav class="evidence-links" aria-label="Evidence for ${html(execution.id)}">
|
||||
${screenshots.map((item) => `<a href="${html(item.href)}" target="_blank" rel="noreferrer">Open ${html(item.label.toLowerCase())}</a>`).join("")}
|
||||
${playwright ? `<a href="${html(playwright)}">Open Playwright report</a>` : ""}
|
||||
${structuredLinks.join("")}
|
||||
</nav>`
|
||||
@@ -302,7 +297,9 @@ function renderCase(
|
||||
class="case case-${state}"
|
||||
data-execution-id="${html(execution.id)}"
|
||||
data-report-case
|
||||
data-report-case-key="${html(`${execution.suite.id}.${execution.task.id}`)}"
|
||||
data-report-suite="${html(execution.suite.id)}"
|
||||
data-report-features="${html(journeyFeatureIds(execution).join(" "))}"
|
||||
data-report-profile="${html(execution.profile.id)}"
|
||||
data-report-environment="${html(execution.environment.id)}"
|
||||
data-report-status="${html(state)}"
|
||||
@@ -312,14 +309,20 @@ function renderCase(
|
||||
<strong>${html(execution.task.label)}</strong>
|
||||
<span class="status">${html(label)}</span>
|
||||
</div>
|
||||
${entry ? renderCaseOutcome(entry.result, entry.valid, entry.errors) : ""}
|
||||
${features.length ? `<div class="case-features">${features.map((feature) =>
|
||||
`<button type="button" class="feature-choice" data-report-feature-choice="${html(feature.id)}" title="${html(capabilities.filter((capability) => capability.feature.id === feature.id).map((capability) => capability.id).join(", "))}">${html(feature.title)}</button>`,
|
||||
).join("")}</div>` : ""}
|
||||
${passed ? "" : outcome}
|
||||
${!entry ? `<p class="case-empty">No result artifact was uploaded.</p>` : ""}
|
||||
${gallery}
|
||||
<details class="case-context case-diagnostics">
|
||||
<summary><span>Test details</span><span class="case-detail-summary">${matcherResults.length ? `${passedMatchers}/${matcherResults.length} checks${notReached ? ` · ${notReached} not reached` : ""}` : "No checks recorded"}${entry ? ` · ${html(durationLabel(entry.result.durationMs))}` : ""}</span></summary>
|
||||
${passed ? outcome : ""}
|
||||
${billingStrip}
|
||||
${links}
|
||||
${renderFirstTaskDetails(entry?.result)}
|
||||
<code class="execution-id">${html(execution.id)}</code>
|
||||
<details class="case-context">
|
||||
<summary>Matchers and test context</summary>
|
||||
${capabilities.length ? `<ul class="case-capabilities" aria-label="Capabilities tested">${capabilities.map((capability) => `<li>${html(capability.feature.title)} · ${html(capability.label)}</li>`).join("")}</ul>` : ""}
|
||||
${
|
||||
entry
|
||||
? `<dl>
|
||||
@@ -633,6 +636,10 @@ function renderSuiteMatrix(input: {
|
||||
return `<section class="suite-section" id="suite-${html(suite.id)}">
|
||||
<div class="section-heading"><div><p class="eyebrow">Test suite</p><h2>${html(suite.label)}</h2></div><p>${html(suite.description)}</p></div>
|
||||
${summaryHtml}
|
||||
<nav class="suite-case-toc" aria-label="${html(suite.label)} cases">
|
||||
<div class="suite-case-toc-heading"><h3>Cases</h3><a href="#overview">Back to index ↑</a></div>
|
||||
${renderSuiteCaseIndex(input.suiteCatalog)}
|
||||
</nav>
|
||||
<div class="table-kicker"><strong>Configuration matrix</strong><span>${profiles.length} profiles · ${environments.length} environments · ${selected} selected</span></div>
|
||||
<div class="table-wrap"><table class="matrix"><colgroup><col class="profile-column">${environments.map(() => '<col class="environment-column">').join("")}</colgroup><thead><tr><th scope="col">Agent profile</th>${environmentHeaders}</tr></thead><tbody>${rows}</tbody></table></div>
|
||||
</section>`;
|
||||
@@ -640,12 +647,12 @@ function renderSuiteMatrix(input: {
|
||||
|
||||
|
||||
export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
const expected = new Set(input.expected);
|
||||
const catalog = discoverReportCatalog({
|
||||
catalog: input.catalog,
|
||||
expected: input.expected,
|
||||
results: input.entries.map((entry) => entry.result),
|
||||
});
|
||||
const expected = new Set(input.expected);
|
||||
}).filter((execution) => expected.has(execution.id));
|
||||
const entryById = new Map(
|
||||
input.entries.map((entry) => [entry.result.executionId, entry]),
|
||||
);
|
||||
@@ -688,9 +695,6 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
)
|
||||
.join("");
|
||||
const historySection = renderHistory(input.history);
|
||||
const publicSummaryImageHref = safePublicAssetHref(
|
||||
input.publicSummaryImageHref,
|
||||
);
|
||||
const filterProfiles = [
|
||||
...new Map(
|
||||
catalog.map((execution) => [execution.profile.id, execution.profile]),
|
||||
@@ -748,7 +752,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
--chat-human-foreground: #ffffff;
|
||||
--chat-bubble-radius: 1rem;
|
||||
--chat-body-size: 0.875rem;
|
||||
--navigation-sticky-offset: 58px;
|
||||
--navigation-sticky-offset: 0px;
|
||||
--font-sans: "Paperclip Inter", Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
--font-mono: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", monospace;
|
||||
}
|
||||
@@ -780,6 +784,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
}
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
[hidden] { display: none !important; }
|
||||
html { scroll-behavior: smooth; }
|
||||
body { margin: 0; background: var(--background); color: var(--foreground); font: 14px/1.5 var(--font-sans); font-feature-settings: "ss01", "cv11"; }
|
||||
button, summary, a { -webkit-tap-highlight-color: transparent; }
|
||||
@@ -805,7 +810,9 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.gallery-launch { min-width: 138px; padding: 10px 15px; }
|
||||
.gallery-launch:hover, .gallery-control:hover, .gallery-close:hover { border-color: var(--foreground); background: var(--primary); color: var(--primary-foreground); }
|
||||
.gallery-launch:focus-visible, .gallery-control:focus-visible, .gallery-close:focus-visible, .screenshot-trigger:focus-visible, .report-filters input:focus-visible, .report-filters select:focus-visible, .filter-reset:focus-visible, summary:focus-visible, a:focus-visible { outline: 2px solid #2563eb; outline-offset: 3px; }
|
||||
.report-filters { display: grid; grid-template-columns: minmax(240px, 1.6fr) repeat(4, minmax(130px, .7fr)) auto; align-items: end; gap: 10px; margin: 0 0 40px; padding: 12px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--card); }
|
||||
.report-filters { display: grid; grid-template-columns: repeat(6, minmax(0, 1fr)); align-items: end; gap: 10px; margin: 0 0 40px; padding: 12px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--card); }
|
||||
.report-search { grid-column: span 5; }
|
||||
.report-feature { grid-column: span 2; }
|
||||
.report-filters label { display: grid; gap: 5px; min-width: 0; }
|
||||
.report-filters label > span { color: var(--muted-foreground); font-size: 10px; font-weight: 600; letter-spacing: .05em; text-transform: uppercase; }
|
||||
.report-filters input, .report-filters select, .filter-reset { width: 100%; min-height: 38px; padding: 7px 9px; border: 1px solid var(--border-strong); border-radius: calc(var(--radius) * .8); background: var(--background); color: var(--foreground); }
|
||||
@@ -847,9 +854,10 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.agent-10 { background: linear-gradient(to bottom, #f2d95f, #4fbcba); }
|
||||
.generation { display: inline-block; margin-left: 7px; padding: 2px 7px; border: 1px solid var(--border); border-radius: 999px; color: var(--muted-foreground); font: 500 10px/1.4 var(--font-mono); letter-spacing: .06em; text-transform: uppercase; }
|
||||
.case-stack { display: grid; gap: 12px; }
|
||||
.case { min-width: 0; max-width: 100%; padding: 16px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--card); overflow: hidden; }
|
||||
.case { min-width: 0; max-width: 100%; padding: 16px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--card); overflow: hidden; scroll-margin-top: 80px; }
|
||||
.case-heading { display: flex; align-items: start; justify-content: space-between; gap: 12px; }
|
||||
.case-heading strong { font-size: 14px; font-weight: 650; }
|
||||
.case-empty { margin: 12px 0; color: var(--muted-foreground); font-size: 12px; }
|
||||
.status, .matcher-state { display: inline-block; flex: none; padding: 3px 8px; border: 1px solid; border-radius: 999px; font: 600 10px/1.4 var(--font-sans); letter-spacing: .05em; text-transform: uppercase; }
|
||||
.case-passed .status, .matcher-passed .matcher-state { border-color: var(--pass-border); background: var(--pass-bg); color: var(--pass-text); }
|
||||
.case-failed .status, .matcher-failed .matcher-state { border-color: var(--fail-border); background: var(--fail-bg); color: var(--fail-text); }
|
||||
@@ -899,9 +907,9 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.billing-metric strong { display: block; overflow-wrap: anywhere; font: 550 17px/1.25 var(--font-mono); font-variant-numeric: tabular-nums; }
|
||||
.billing-metric span { display: block; margin-top: 4px; color: var(--muted-foreground); font-size: 10px; letter-spacing: .06em; text-transform: uppercase; }
|
||||
.billing-note { grid-column: 1 / -1; margin: 0; padding: 11px 16px; border-top: 1px solid var(--border); color: var(--muted-foreground); font-size: 11px; }
|
||||
.suite-nav { position: sticky; top: 0; z-index: 12; display: flex; flex-wrap: wrap; gap: 6px; margin: 0 0 24px; padding: 10px 0; border-top: 1px solid var(--border); border-bottom: 1px solid var(--border); background: var(--background); }
|
||||
.suite-nav a { padding: 7px 10px; border-radius: calc(var(--radius) * .8); color: var(--muted-foreground); font-size: 12px; font-weight: 600; text-decoration: none; }
|
||||
.suite-nav a:hover { background: var(--raised); color: var(--foreground); }
|
||||
.results-heading { display: flex; align-items: baseline; justify-content: space-between; gap: 16px; margin: 40px 0 20px; }
|
||||
.results-heading h2 { margin: 0; font-size: 24px; }
|
||||
.results-heading a { color: var(--muted-foreground); font-size: 12px; }
|
||||
.suite-section, .history-section { scroll-margin-top: calc(var(--navigation-sticky-offset) + 16px); margin-top: 64px; }
|
||||
.section-heading { display: grid; grid-template-columns: minmax(0, 1fr) minmax(260px, 520px); align-items: end; gap: 32px; margin-bottom: 24px; }
|
||||
.section-heading h2 { margin: 0; font-size: clamp(25px, 3vw, 38px); line-height: 1.05; letter-spacing: -.035em; }
|
||||
@@ -913,6 +921,10 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.suite-summary span, .suite-summary small { color: var(--muted-foreground); font-size: 9px; }
|
||||
.suite-summary span { letter-spacing: .06em; text-transform: uppercase; }
|
||||
.suite-summary strong { margin: 3px 0 1px; font: 550 15px/1.3 var(--font-mono); }
|
||||
.suite-case-toc { margin: 0 0 24px; }
|
||||
.suite-case-toc-heading { display: flex; align-items: baseline; justify-content: space-between; gap: 16px; margin-bottom: 8px; }
|
||||
.suite-case-toc-heading h3 { margin: 0; font-size: 13px; font-weight: 600; }
|
||||
.suite-case-toc-heading > a { color: var(--muted-foreground); font-size: 11px; }
|
||||
.history-pointers { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 12px; margin-bottom: 16px; }
|
||||
.history-pointers a { padding: 14px 16px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--card); text-decoration: none; }
|
||||
.history-pointers span, .history-pointers strong { display: block; }
|
||||
@@ -948,7 +960,16 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.history-failed { border-color: var(--fail-border); background: var(--fail-bg); color: var(--fail-text); }
|
||||
.history-empty { padding: 24px 0; color: var(--muted-foreground); text-align: center; }
|
||||
.case-context > summary, .usage > summary { width: fit-content; cursor: pointer; color: var(--foreground); font-size: 12px; font-weight: 600; }
|
||||
.case-diagnostics { margin-top: 14px; padding-top: 12px; border-top: 1px solid var(--border); }
|
||||
.case-diagnostics > summary { display: flex; align-items: baseline; flex-wrap: wrap; gap: 4px 8px; width: 100%; list-style: none; }
|
||||
.case-diagnostics > summary::-webkit-details-marker { display: none; }
|
||||
.case-diagnostics > summary::before { content: "›"; color: var(--muted-foreground); font-size: 16px; line-height: 1; }
|
||||
.case-diagnostics[open] > summary::before { transform: rotate(90deg); }
|
||||
.case-detail-summary { margin-left: auto; color: var(--muted-foreground); font: 400 11px/1.5 var(--font-mono); }
|
||||
.case-capabilities { padding-left: 16px; color: var(--muted-foreground); font-size: 11px; }
|
||||
.case-outcome { margin: 12px 0; padding: 12px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--raised); font-size: 12px; }
|
||||
.case-incomplete > .case-outcome { border-color: var(--missing-border); }
|
||||
.case-failed > .case-outcome { border-color: var(--fail-border); }
|
||||
.case-outcome p { margin: 8px 0; }
|
||||
.failure-reason { white-space: pre-wrap; overflow-wrap: anywhere; font: 11px/1.6 var(--font-mono); }
|
||||
.conversation-details { margin: 16px 0; }
|
||||
@@ -1033,6 +1054,8 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.gallery-previous { left: 24px; }
|
||||
.gallery-next { right: 24px; }
|
||||
.gallery-position { color: #b8b8b5; font: 12px/1.4 var(--font-mono); font-variant-numeric: tabular-nums; }
|
||||
.gallery-original { color: #fafafa; font-size: 12px; }
|
||||
.back-to-top { position: fixed; right: max(24px, env(safe-area-inset-right)); bottom: max(24px, env(safe-area-inset-bottom)); z-index: 10; display: inline-flex; align-items: center; gap: 8px; min-height: 44px; padding: 10px 16px; border: 1px solid var(--border-strong); border-radius: 999px; background: var(--foreground); color: var(--background); font: 600 12px/1.4 var(--font-sans); text-decoration: none; }
|
||||
@media (max-width: 1180px) {
|
||||
.matrix { display: block; min-width: 0; }
|
||||
.matrix thead { display: none; }
|
||||
@@ -1063,8 +1086,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.trend-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
||||
.history-filters { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
||||
.report-filters { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
||||
.report-search, .filter-result { grid-column: 1 / -1; }
|
||||
.suite-nav { top: 0; }
|
||||
.report-search, .report-feature, .filter-result { grid-column: 1 / -1; }
|
||||
.suite-section, .history-section { scroll-margin-top: 72px; }
|
||||
}
|
||||
@media (max-width: 640px) {
|
||||
@@ -1085,11 +1107,9 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
.gallery-control { width: 40px; height: 56px; }
|
||||
.gallery-previous { left: 8px; }
|
||||
.gallery-next { right: 8px; }
|
||||
.suite-nav { top: 0; overflow-x: auto; flex-wrap: nowrap; }
|
||||
.suite-nav a { white-space: nowrap; }
|
||||
.history-pointers, .trend-grid, .history-filters { grid-template-columns: 1fr; }
|
||||
.report-filters { grid-template-columns: 1fr; }
|
||||
.report-search, .filter-result { grid-column: auto; }
|
||||
.report-search, .report-feature, .filter-result { grid-column: auto; }
|
||||
.suite-summary { grid-template-columns: 1fr; }
|
||||
.suite-summary > div, .suite-summary > div:nth-child(even) { border-right: 0; border-bottom: 1px solid var(--border); }
|
||||
.suite-summary > div:last-child { border-bottom: 0; }
|
||||
@@ -1098,8 +1118,8 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
*, *::before, *::after { scroll-behavior: auto !important; transition-duration: 100ms !important; }
|
||||
}
|
||||
.public-summary { margin: 0 0 32px; padding: 12px; border: 1px solid var(--border); border-radius: 10px; background: var(--raised); }
|
||||
.public-summary img { display: block; width: 100%; height: auto; border-radius: 6px; }
|
||||
${campaignSuiteSummaryStyles}
|
||||
${journeyFeatureStyles}
|
||||
@media print {
|
||||
.brand-bar, .gallery-launch, dialog { display: none; }
|
||||
main { width: 100%; margin: 0; }
|
||||
@@ -1108,7 +1128,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<body id="top">
|
||||
<div class="brand-bar">
|
||||
<a class="brand-lockup" href="https://paperclip.ing" aria-label="Paperclip home">
|
||||
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="m16 6-8.414 8.586a2 2 0 0 0 2.829 2.829l8.414-8.586a4 4 0 1 0-5.657-5.657l-8.379 8.551a6 6 0 1 0 8.485 8.485l8.379-8.551"/></svg>
|
||||
@@ -1120,7 +1140,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
<header class="report-header">
|
||||
<div>
|
||||
<p class="eyebrow">Full-stack acceptance campaign</p>
|
||||
<h1>${html(input.title)}</h1>
|
||||
<h1 tabindex="-1">${html(input.title)}</h1>
|
||||
<p class="lede">A browser-verified matrix of runner profiles, execution environments, and deterministic task contracts. Declared PNG screenshots and normalized results are retained with every published campaign; additional diagnostic evidence remains in the access-controlled workflow artifact.</p>
|
||||
</div>
|
||||
<div class="report-actions">
|
||||
@@ -1133,7 +1153,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
<button class="gallery-launch" type="button" data-gallery-open ${screenshotCount === 0 ? "disabled" : ""}>${screenshotCount === 0 ? "Visual evidence · workflow artifact only" : `View gallery · ${screenshotCount}`}</button>
|
||||
</div>
|
||||
</header>
|
||||
${publicSummaryImageHref ? `<figure class="public-summary"><img src="${html(publicSummaryImageHref)}" alt="Runner E2E campaign status summary"></figure>` : ""}
|
||||
${renderCampaignSuiteSummary(input.campaign?.suites ?? [], catalog, "", true)}
|
||||
<section class="billing-overview" aria-label="Campaign billing summary">
|
||||
<div class="billing-metric"><strong>${html(tokenLabel(campaignBilling.llm.inputTokens, campaignBilling.llm))}</strong><span>Input tokens</span></div>
|
||||
<div class="billing-metric"><strong>${html(tokenLabel(campaignBilling.llm.outputTokens, campaignBilling.llm))}</strong><span>Output tokens</span></div>
|
||||
@@ -1147,24 +1167,23 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
<div class="billing-metric"><strong>${campaignBilling.llm.runsWithReportedCost}/${campaignBilling.llm.runCount}</strong><span>Runs provider-priced</span></div>
|
||||
<p class="billing-note">Model spend is the provider-reported subtotal; unpriced or unavailable runs are excluded, never counted as free. Daytona runtime is a public-list-price estimate from captured lease time and pinned resources, before credits, discounts, storage allowance, or invoice adjustments. Local execution has no external runtime meter.</p>
|
||||
</section>
|
||||
<nav class="suite-nav" aria-label="Report sections">
|
||||
<a href="#overview">Overview</a>
|
||||
${suites.map((suite) => `<a href="#suite-${html(suite.id)}">${html(suite.label)}</a>`).join("")}
|
||||
<a href="#history">History</a>
|
||||
</nav>
|
||||
<div class="results-heading" id="results"><h2>Results</h2><a href="#history">Campaign history</a></div>
|
||||
${renderJourneyFeatureCoverage(catalog, entryById)}
|
||||
<section class="report-filters" aria-label="Filter test results">
|
||||
<label class="report-search"><span>Search tests</span><input type="search" placeholder="Test, model, provider, execution ID…" autocomplete="off" data-report-query></label>
|
||||
<button class="filter-reset" type="button" data-report-reset>Reset</button>
|
||||
<label class="report-feature"><span>Feature</span><select data-report-feature><option value="">All features</option>${renderJourneyFeatureOptions(catalog)}</select></label>
|
||||
<label><span>Agent profile</span><select data-report-profile><option value="">All profiles</option>${filterProfiles.map((profile) => `<option value="${html(profile.id)}">${html(profile.label)}</option>`).join("")}</select></label>
|
||||
<label><span>Environment</span><select data-report-environment><option value="">All environments</option>${filterEnvironments.map((environment) => `<option value="${html(environment.id)}">${html(environment.label)}</option>`).join("")}</select></label>
|
||||
<label><span>Suite</span><select data-report-suite><option value="">All suites</option>${suites.map((suite) => `<option value="${html(suite.id)}">${html(suite.label)}</option>`).join("")}</select></label>
|
||||
<label><span>Status</span><select data-report-status><option value="">All statuses</option><option value="passed">Passed</option><option value="failed">Failed</option><option value="incomplete">Incomplete</option><option value="missing">Missing</option><option value="not-selected">Not selected</option></select></label>
|
||||
<button class="filter-reset" type="button" data-report-reset>Reset</button>
|
||||
<label><span>Status</span><select data-report-status><option value="">All statuses</option><option value="passed">Passed</option><option value="failed">Failed</option><option value="incomplete">Incomplete</option><option value="missing">Missing</option></select></label>
|
||||
<p class="filter-result" data-report-result aria-live="polite"></p>
|
||||
</section>
|
||||
${suiteSections}
|
||||
${historySection}
|
||||
<footer><span>Generated ${html(input.generatedAt)}</span><span>${catalog.length} catalog executions · Declared screenshots and normalized results published</span></footer>
|
||||
<footer><span>Generated ${html(input.generatedAt)}</span><span>${catalog.length} selected executions · Declared screenshots and normalized results published</span></footer>
|
||||
</main>
|
||||
<a class="back-to-top" href="#top" data-back-to-top hidden><span aria-hidden="true">↑</span>Back to top</a>
|
||||
<dialog class="gallery-dialog" data-gallery-dialog aria-labelledby="gallery-title">
|
||||
<div class="gallery-shell">
|
||||
<div class="gallery-toolbar">
|
||||
@@ -1192,10 +1211,26 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
</div>
|
||||
<div class="gallery-footer">
|
||||
<span class="gallery-position" data-gallery-position aria-live="polite"></span>
|
||||
<a class="gallery-original" data-gallery-original target="_blank" rel="noreferrer">Open original image ↗</a>
|
||||
</div>
|
||||
</div>
|
||||
</dialog>
|
||||
<script>
|
||||
(() => {
|
||||
const backToTop = document.querySelector("[data-back-to-top]");
|
||||
const update = () => { backToTop.hidden = window.scrollY < window.innerHeight; };
|
||||
window.addEventListener("scroll", update, { passive: true });
|
||||
window.addEventListener("resize", update);
|
||||
backToTop.addEventListener("click", (event) => {
|
||||
if (event.button !== 0 || event.metaKey || event.ctrlKey || event.shiftKey || event.altKey) return;
|
||||
event.preventDefault();
|
||||
window.scrollTo({ top: 0, behavior: "instant" });
|
||||
if (location.hash !== "#top") history.pushState(null, "", "#top");
|
||||
document.querySelector(".report-header h1").focus({ preventScroll: true });
|
||||
update();
|
||||
});
|
||||
update();
|
||||
})();
|
||||
document.addEventListener("click", (event) => {
|
||||
const link = event.target.closest("a[href^='#first-task-']");
|
||||
if (!link) return;
|
||||
@@ -1223,6 +1258,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
const tokens = dialog.querySelector("[data-gallery-tokens]");
|
||||
const matchers = dialog.querySelector("[data-gallery-matchers]");
|
||||
const position = dialog.querySelector("[data-gallery-position]");
|
||||
const original = dialog.querySelector("[data-gallery-original]");
|
||||
const stage = dialog.querySelector("[data-gallery-stage]");
|
||||
let activeItem = items[0];
|
||||
let pointerStartX = null;
|
||||
@@ -1237,6 +1273,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
}
|
||||
const item = activeItem;
|
||||
image.src = item.dataset.galleryHref;
|
||||
original.href = item.dataset.galleryHref;
|
||||
image.alt = item.dataset.galleryLabel + " for " + item.dataset.galleryExecution;
|
||||
title.textContent = item.dataset.galleryLabel;
|
||||
execution.textContent = item.dataset.galleryExecution;
|
||||
@@ -1298,6 +1335,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
const cases = Array.from(document.querySelectorAll("[data-report-case]"));
|
||||
if (cases.length === 0) return;
|
||||
const query = document.querySelector("[data-report-query]");
|
||||
const feature = document.querySelector("[data-report-feature]");
|
||||
const profile = document.querySelector("[data-report-profile]");
|
||||
const environment = document.querySelector("[data-report-environment]");
|
||||
const suite = document.querySelector("[data-report-suite]");
|
||||
@@ -1305,12 +1343,20 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
const reset = document.querySelector("[data-report-reset]");
|
||||
const result = document.querySelector("[data-report-result]");
|
||||
const galleryLaunch = document.querySelector("[data-gallery-open]");
|
||||
const caseLinks = Array.from(document.querySelectorAll("[data-report-case-link]"));
|
||||
const casesByKey = new Map();
|
||||
cases.forEach((item) => {
|
||||
const key = item.dataset.reportCaseKey;
|
||||
if (!casesByKey.has(key)) casesByKey.set(key, []);
|
||||
casesByKey.get(key).push(item);
|
||||
});
|
||||
const apply = () => {
|
||||
const search = query.value.trim().toLowerCase();
|
||||
let visible = 0;
|
||||
cases.forEach((item) => {
|
||||
const matches =
|
||||
(!search || item.dataset.reportSearch.includes(search)) &&
|
||||
(!feature.value || (feature.value === "__unmapped__" ? !item.dataset.reportFeatures : item.dataset.reportFeatures.split(" ").includes(feature.value))) &&
|
||||
(!profile.value || item.dataset.reportProfile === profile.value) &&
|
||||
(!environment.value || item.dataset.reportEnvironment === environment.value) &&
|
||||
(!suite.value || item.dataset.reportSuite === suite.value) &&
|
||||
@@ -1324,6 +1370,26 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
document.querySelectorAll(".suite-section").forEach((section) => {
|
||||
section.hidden = !section.querySelector("[data-report-case]:not([hidden])");
|
||||
});
|
||||
caseLinks.forEach((link) => {
|
||||
const variants = casesByKey.get(link.dataset.reportCaseLink);
|
||||
const matching = variants.filter((item) => !item.hidden);
|
||||
const selection = matching.length ? matching : variants;
|
||||
const counts = { passed: 0, failed: 0, incomplete: 0, missing: 0 };
|
||||
selection.forEach((item) => { counts[item.dataset.reportStatus] += 1; });
|
||||
const state = counts.failed ? "failed" : counts.incomplete ? "incomplete" : counts.missing ? "missing" : "passed";
|
||||
const label = { passed: "Passed", failed: "Failed", incomplete: "Incomplete", missing: "Missing result" }[state];
|
||||
const context = matching.length === variants.length ? "All configurations" : matching.length ? "Matching configurations" : "All configurations (none match current filters)";
|
||||
const description = label + " · " + counts.passed + "/" + selection.length + " passed"
|
||||
+ (counts.failed ? " · " + counts.failed + " failed" : "")
|
||||
+ (counts.incomplete ? " · " + counts.incomplete + " incomplete" : "")
|
||||
+ (counts.missing ? " · " + counts.missing + " missing" : "")
|
||||
+ " — " + context;
|
||||
const target = selection.find((item) => item.dataset.reportStatus === state) || selection[0];
|
||||
link.dataset.caseStatus = state;
|
||||
link.setAttribute("title", description);
|
||||
link.setAttribute("aria-description", description);
|
||||
link.setAttribute("href", "#execution-" + target.dataset.executionId);
|
||||
});
|
||||
const visibleImages = cases.reduce(
|
||||
(total, item) => total + (item.hidden ? 0 : item.querySelectorAll("[data-gallery-item]").length),
|
||||
0,
|
||||
@@ -1334,18 +1400,49 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
galleryLaunch.textContent = visibleImages === 0 ? "No visual evidence in selection" : "View gallery · " + visibleImages;
|
||||
}
|
||||
};
|
||||
[query, profile, environment, suite, status].forEach((control) => {
|
||||
const controls = [query, feature, profile, environment, suite, status];
|
||||
const clearFilters = () => {
|
||||
controls.forEach((control) => { control.value = ""; });
|
||||
};
|
||||
const revealTarget = (hash) => {
|
||||
if (!hash.startsWith("#suite-") && !hash.startsWith("#execution-")) return false;
|
||||
const target = document.getElementById(hash.slice(1));
|
||||
if (!target || !target.matches(".suite-section, [data-report-case]")) return false;
|
||||
if (target.hidden) {
|
||||
clearFilters();
|
||||
apply();
|
||||
}
|
||||
requestAnimationFrame(() => {
|
||||
target.scrollIntoView({ block: "start", behavior: "instant" });
|
||||
});
|
||||
return true;
|
||||
};
|
||||
document.querySelectorAll('a[href^="#suite-"], a[href^="#execution-"]').forEach((link) => {
|
||||
link.addEventListener("click", (event) => {
|
||||
if (event.defaultPrevented || event.button !== 0 || event.metaKey || event.ctrlKey || event.shiftKey || event.altKey) return;
|
||||
const hash = link.getAttribute("href");
|
||||
if (!revealTarget(hash)) return;
|
||||
event.preventDefault();
|
||||
if (location.hash !== hash) history.pushState(null, "", hash);
|
||||
});
|
||||
});
|
||||
window.addEventListener("hashchange", () => { revealTarget(location.hash); });
|
||||
controls.forEach((control) => {
|
||||
control.addEventListener(control === query ? "input" : "change", apply);
|
||||
});
|
||||
reset.addEventListener("click", () => {
|
||||
query.value = "";
|
||||
profile.value = "";
|
||||
environment.value = "";
|
||||
suite.value = "";
|
||||
status.value = "";
|
||||
clearFilters();
|
||||
apply();
|
||||
query.focus();
|
||||
});
|
||||
document.querySelectorAll("[data-report-feature-choice]").forEach((button) => {
|
||||
button.addEventListener("click", () => {
|
||||
feature.value = button.dataset.reportFeatureChoice;
|
||||
apply();
|
||||
document.querySelector(".feature-coverage").open = false;
|
||||
document.getElementById("results").scrollIntoView({ block: "start" });
|
||||
});
|
||||
});
|
||||
document.addEventListener("keydown", (event) => {
|
||||
if (event.key !== "/" || event.metaKey || event.ctrlKey || event.altKey) return;
|
||||
const target = event.target;
|
||||
@@ -1354,6 +1451,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
query.focus();
|
||||
});
|
||||
apply();
|
||||
revealTarget(location.hash);
|
||||
})();
|
||||
</script>
|
||||
<script>
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
import { journeyCapabilities, journeyFeatureIds, journeyFeatureMap, journeyFeatures } from "./journey-features.js";
|
||||
import type { ReportExecution } from "./report-catalog.js";
|
||||
|
||||
function html(value: unknown) {
|
||||
return String(value ?? "").replace(/[&<>"']/g, (character) =>
|
||||
({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" })[character]!,
|
||||
);
|
||||
}
|
||||
|
||||
export function renderJourneyFeatureOptions(catalog: readonly ReportExecution[]) {
|
||||
const selected = new Set(catalog.flatMap(journeyFeatureIds));
|
||||
return journeyFeatureMap.sections.map((section) => {
|
||||
const features = section.features.filter((feature) => selected.has(feature.id));
|
||||
return features.length ? `<optgroup label="${html(section.title)}">${features.map((feature) =>
|
||||
`<option value="${html(feature.id)}">${html(feature.title)}</option>`,
|
||||
).join("")}</optgroup>` : "";
|
||||
}).join("") + (catalog.some((execution) => journeyCapabilities(execution).length === 0)
|
||||
? '<option value="__unmapped__">Unmapped journeys</option>' : "");
|
||||
}
|
||||
|
||||
export function renderJourneyFeatureCoverage(
|
||||
catalog: readonly ReportExecution[],
|
||||
entries: ReadonlyMap<string, { valid: boolean }>,
|
||||
) {
|
||||
const linked = new Set(catalog.flatMap((execution) => journeyCapabilities(execution).map((capability) => capability.id)));
|
||||
const mappedTests = catalog.filter((execution) => journeyCapabilities(execution).length > 0).length;
|
||||
const totalCapabilities = journeyFeatures.reduce((total, feature) => total + feature.capabilities.length, 0);
|
||||
return `<details class="feature-coverage" id="feature-coverage">
|
||||
<summary>Feature coverage <span>${linked.size} of ${totalCapabilities} capabilities linked · ${mappedTests} of ${catalog.length} tests mapped</span></summary>
|
||||
<p>Associations use the current feature map. Counts describe linked test scenarios; a passing journey does not establish full coverage of a capability.</p>
|
||||
${mappedTests < catalog.length ? `<button type="button" class="feature-choice" data-report-feature-choice="__unmapped__">Find ${catalog.length - mappedTests} unmapped tests</button>` : ""}
|
||||
<div class="feature-coverage-sections">${journeyFeatureMap.sections.map((section) => `<section>
|
||||
<h3>${html(section.title)}</h3>
|
||||
<table><thead><tr><th>Feature</th><th>Capabilities linked</th><th>Linked test results</th></tr></thead><tbody>${section.features.map((feature) => {
|
||||
const tests = catalog.filter((execution) => journeyFeatureIds(execution).includes(feature.id));
|
||||
const mapped = feature.capabilities.filter((capability) => linked.has(capability.id));
|
||||
const passed = tests.filter((execution) => entries.get(execution.id)?.valid).length;
|
||||
const missing = tests.filter((execution) => !entries.has(execution.id)).length;
|
||||
return `<tr><th scope="row">${tests.length
|
||||
? `<button type="button" class="feature-choice" data-report-feature-choice="${html(feature.id)}">${html(feature.title)}</button>`
|
||||
: html(feature.title)}<small>${mapped.map((capability) => html(capability.label)).join(" · ")}</small></th>
|
||||
<td>${mapped.length}/${feature.capabilities.length}</td>
|
||||
<td>${tests.length ? `${passed}/${tests.length} passed${tests.length - passed - missing ? ` · ${tests.length - passed - missing} failed or incomplete` : ""}${missing ? ` · ${missing} missing` : ""}` : "No mapped journey"}</td></tr>`;
|
||||
}).join("")}</tbody></table>
|
||||
</section>`).join("")}</div>
|
||||
</details>`;
|
||||
}
|
||||
|
||||
export const journeyFeatureStyles = `
|
||||
.feature-coverage { margin: 0 0 24px; padding: 14px 0; border-top: 1px solid var(--border); border-bottom: 1px solid var(--border); }
|
||||
.feature-coverage > summary { cursor: pointer; font-weight: 600; }
|
||||
.feature-coverage > summary span { margin-left: 12px; color: var(--muted-foreground); font-size: 12px; font-weight: 400; }
|
||||
.feature-coverage > p { max-width: 780px; color: var(--muted-foreground); font-size: 12px; }
|
||||
.feature-coverage-sections { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 24px; margin-top: 24px; }
|
||||
.feature-coverage h3 { margin: 0 0 12px; font-size: 14px; }
|
||||
.feature-coverage table { width: 100%; border-collapse: collapse; font-size: 11px; }
|
||||
.feature-coverage th, .feature-coverage td { padding: 10px 8px; border-bottom: 1px solid var(--border); text-align: left; vertical-align: top; }
|
||||
.feature-coverage thead th { color: var(--muted-foreground); font-weight: 500; }
|
||||
.feature-coverage th small { display: block; margin-top: 4px; color: var(--muted-foreground); font-weight: 400; }
|
||||
.feature-choice { padding: 0; border: 0; background: transparent; color: inherit; cursor: pointer; font: inherit; text-align: left; text-decoration: underline; text-underline-offset: 3px; }
|
||||
.case-features { display: flex; flex-wrap: wrap; gap: 4px 12px; margin: 8px 0; color: var(--muted-foreground); font-size: 10px; }
|
||||
@media (max-width: 860px) { .feature-coverage-sections { grid-template-columns: 1fr; } .feature-coverage > summary span { display: block; margin: 4px 0 0; } }
|
||||
`;
|
||||
@@ -304,9 +304,10 @@ describe("first-task conversation report", () => {
|
||||
expect(rendered).not.toContain(
|
||||
'href="evidence/codex/html-report/index.html"',
|
||||
);
|
||||
expect(rendered.indexOf("Full run / tool-event JSON")).toBeLessThan(
|
||||
rendered.indexOf("Matchers and test context</summary>"),
|
||||
);
|
||||
const diagnostics = rendered.indexOf('class="case-context case-diagnostics"');
|
||||
expect(diagnostics).toBeGreaterThan(0);
|
||||
expect(rendered.indexOf("Credential-persistence check failed")).toBeLessThan(diagnostics);
|
||||
expect(rendered.indexOf("Full run / tool-event JSON")).toBeGreaterThan(diagnostics);
|
||||
expect(rendered).toContain(
|
||||
'id="first-task-first-task.legacy-codex.local.clear-task-first-response-response"',
|
||||
);
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { campaignSuiteSummaryStyles, renderCampaignSuiteSummary } from "./campaign-summary.js";
|
||||
import type {
|
||||
RunnerE2EHistoryCampaign,
|
||||
RunnerE2EHistoryIndex,
|
||||
@@ -37,17 +39,6 @@ function date(value: string) {
|
||||
}).format(new Date(value));
|
||||
}
|
||||
|
||||
function safeRelativeAssetHref(relative: string | undefined) {
|
||||
if (!relative || /^(?:[a-z]+:|\/\/|\/)/i.test(relative)) return null;
|
||||
const segments = relative.split("/");
|
||||
if (
|
||||
segments.some((segment) => !segment || segment === "." || segment === "..")
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
return segments.map(encodeURIComponent).join("/");
|
||||
}
|
||||
|
||||
function campaignStatus(campaign: RunnerE2EHistoryCampaign) {
|
||||
if (campaign.failed === 0 && (campaign.incomplete ?? 0) > 0) return "incomplete";
|
||||
return campaign.failed === 0 &&
|
||||
@@ -112,7 +103,6 @@ function campaignRow(campaign: RunnerE2EHistoryCampaign) {
|
||||
|
||||
export function renderRunnerHistoryIndex(
|
||||
history: RunnerE2EHistoryIndex,
|
||||
options: { latestSummaryImageHref?: string } = {},
|
||||
) {
|
||||
const campaigns = [...history.campaigns].sort((left, right) =>
|
||||
right.generatedAt.localeCompare(left.generatedAt),
|
||||
@@ -134,9 +124,6 @@ export function renderRunnerHistoryIndex(
|
||||
campaigns.length > 0
|
||||
? campaigns.map(campaignRow).join("")
|
||||
: `<tr><td class="empty" colspan="9">No campaigns have been published yet.</td></tr>`;
|
||||
const latestSummaryImageHref = safeRelativeAssetHref(
|
||||
options.latestSummaryImageHref,
|
||||
);
|
||||
|
||||
return `<!doctype html>
|
||||
<html lang="en">
|
||||
@@ -173,8 +160,13 @@ export function renderRunnerHistoryIndex(
|
||||
.pointers { display:flex; flex-wrap:wrap; gap:10px; margin-bottom:16px; }
|
||||
.pointers a { padding:8px 11px; border:1px solid var(--border); border-radius:7px; background:var(--raised); font-size:12px; text-decoration:none; }
|
||||
.pointers a:hover,.campaign-link:hover,.open-cell a:hover { text-decoration:underline; }
|
||||
.latest-summary { margin:0 0 24px; padding:12px; border:1px solid var(--border); border-radius:10px; background:var(--raised); }
|
||||
.latest-summary img { display:block; width:100%; height:auto; border-radius:6px; }
|
||||
.latest-campaign { margin-bottom:32px; }
|
||||
.latest-campaign-heading { display:flex; flex-wrap:wrap; align-items:baseline; gap:12px; margin-bottom:20px; }
|
||||
.latest-campaign-heading h2 { margin:0; font-size:18px; font-weight:600; }
|
||||
.latest-metrics { display:flex; flex-wrap:wrap; gap:24px 48px; margin:0 0 16px; }
|
||||
.latest-metrics dt { color:var(--muted); font-size:11px; }
|
||||
.latest-metrics dd { margin:4px 0 0; font:600 24px/1.2 var(--mono); }
|
||||
${campaignSuiteSummaryStyles}
|
||||
.table-wrap { border-top:1px solid var(--border); border-bottom:1px solid var(--border); }
|
||||
table { width:100%; border-collapse:collapse; }
|
||||
th,td { padding:14px 12px; border-bottom:1px solid var(--border); text-align:left; vertical-align:top; }
|
||||
@@ -226,7 +218,15 @@ export function renderRunnerHistoryIndex(
|
||||
<div class="metric"><strong>${html(usd(totalCost))}</strong><span>Recorded cost</span></div>
|
||||
</div>
|
||||
</header>
|
||||
${latestSummaryImageHref ? `<figure class="latest-summary"><img src="${html(latestSummaryImageHref)}" alt="Latest runner E2E campaign status summary"></figure>` : ""}
|
||||
${latest ? `<section class="latest-campaign" aria-label="Latest campaign summary">
|
||||
<div class="latest-campaign-heading"><h2>Latest campaign</h2><a class="campaign-link" href="${html(latest.publicUrl)}">${html(latest.campaignId)}</a></div>
|
||||
<dl class="latest-metrics">
|
||||
<div><dt>Passed</dt><dd>${latest.passed}/${latest.selected}</dd></div>
|
||||
<div><dt>Failed or incomplete</dt><dd>${latest.failed + (latest.incomplete ?? 0)}</dd></div>
|
||||
<div><dt>Test time</dt><dd>${html(duration(latest.suites.reduce((total, suite) => total + suite.durationMs, 0)))}</dd></div>
|
||||
</dl>
|
||||
${renderCampaignSuiteSummary(latest.suites, runnerMatrix, latest.publicUrl)}
|
||||
</section>` : ""}
|
||||
<nav class="pointers" aria-label="Campaign pointers">
|
||||
${latest ? `<a href="${html(latest.publicUrl)}">Latest run · ${html(latest.campaignId)}</a>` : ""}
|
||||
${latestGreen ? `<a href="${html(latestGreen.publicUrl)}">Latest complete green · ${html(latestGreen.campaignId)}</a>` : ""}
|
||||
|
||||
@@ -567,7 +567,6 @@ async function main() {
|
||||
await regenerateRunnerDashboard({
|
||||
bundle: s3ReportRoot,
|
||||
historyFile: null,
|
||||
publicSummaryImageHref: "public-images/campaign-summary.png",
|
||||
});
|
||||
const manifest = await createBundleManifest(
|
||||
s3ReportRoot,
|
||||
@@ -635,16 +634,10 @@ async function main() {
|
||||
historyFile,
|
||||
outputDirectory: landingDirectory,
|
||||
evidenceHrefPrefix: `campaigns/${campaign.campaignId}`,
|
||||
publicSummaryImageHref: `campaigns/${campaign.campaignId}/public-images/campaign-summary.png`,
|
||||
});
|
||||
await writeFile(
|
||||
path.join(landingDirectory, "index.html"),
|
||||
renderRunnerHistoryIndex(history, {
|
||||
latestSummaryImageHref:
|
||||
history.latestCampaignId === campaign.campaignId
|
||||
? `campaigns/${campaign.campaignId}/public-images/campaign-summary.png`
|
||||
: undefined,
|
||||
}),
|
||||
renderRunnerHistoryIndex(history),
|
||||
"utf8",
|
||||
);
|
||||
await Promise.all([
|
||||
|
||||
@@ -289,19 +289,17 @@ describe("runner E2E campaign history", () => {
|
||||
expect(dashboard).toContain("Suite pass rate");
|
||||
expect(dashboard).toContain("lines break at definition changes");
|
||||
expect(dashboard).toContain("cleanup passed");
|
||||
const index = renderRunnerHistoryIndex(history, {
|
||||
latestSummaryImageHref:
|
||||
"campaigns/complete-red/public-images/campaign-summary.png",
|
||||
});
|
||||
const index = renderRunnerHistoryIndex(history);
|
||||
expect(index).toContain("Runner E2E campaigns");
|
||||
expect(index).toContain("complete-green");
|
||||
expect(index).toContain("complete-red");
|
||||
expect(index).toContain(`${runnerMatrix.length}/${runnerMatrix.length} passed`);
|
||||
expect(index).toContain(`${runnerMatrix.length - 1}/${runnerMatrix.length} passed`);
|
||||
expect(index).toContain("Open report →");
|
||||
expect(index).toContain(
|
||||
"campaigns/complete-red/public-images/campaign-summary.png",
|
||||
);
|
||||
expect(index).toContain('aria-label="Latest campaign summary"');
|
||||
expect(index).toContain('aria-label="Suite results"');
|
||||
expect(index).toContain("Core Runner Compatibility</a>");
|
||||
expect(index).not.toContain("campaign-summary.png");
|
||||
expect(index).toContain(
|
||||
"declared screenshots, and normalized results",
|
||||
);
|
||||
@@ -605,11 +603,11 @@ describe("historical publication security", () => {
|
||||
).toBe(true);
|
||||
await regenerateRunnerDashboard({
|
||||
bundle: root,
|
||||
publicSummaryImageHref: "public-images/campaign-summary.png",
|
||||
});
|
||||
expect(await readFile(path.join(root, "index.html"), "utf8")).toContain(
|
||||
'src="public-images/campaign-summary.png"',
|
||||
);
|
||||
const dashboardHtml = await readFile(path.join(root, "index.html"), "utf8");
|
||||
expect(dashboardHtml).toContain('aria-label="Suite results"');
|
||||
expect(dashboardHtml).toContain(`${execution.suite.label}</a>`);
|
||||
expect(dashboardHtml).not.toContain('src="public-images/campaign-summary.png"');
|
||||
const manifest = await createBundleManifest(
|
||||
root,
|
||||
campaign.campaignId,
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { featureIndex } from "../../scripts/feature-map.mjs";
|
||||
import type { ReportExecution } from "./report-catalog.js";
|
||||
|
||||
export const journeyFeatureMap = featureIndex(fileURLToPath(new URL("../../", import.meta.url)));
|
||||
export const journeyFeatures = journeyFeatureMap.sections.flatMap((section) =>
|
||||
section.features.map((feature) => ({ ...feature, section: section.title })),
|
||||
);
|
||||
const capabilities = new Map(journeyFeatures.flatMap((feature) =>
|
||||
feature.capabilities.map((capability) => [capability.id, { ...capability, feature }] as const),
|
||||
));
|
||||
const associations = new Map<string, readonly string[]>();
|
||||
|
||||
function associate(suites: readonly string[], cases: Record<string, readonly string[]>) {
|
||||
for (const [caseId, ids] of Object.entries(cases)) {
|
||||
for (const id of ids) if (!capabilities.has(id)) throw new Error(`Unknown journey capability: ${id}`);
|
||||
for (const suite of suites) associations.set(`${suite}/${caseId}`, ids);
|
||||
}
|
||||
}
|
||||
|
||||
associate(["core-compatibility", "openrouter-model-breadth", "extended-harnesses", "grok-qualification", "grok-subscription-qualification"], {
|
||||
"message-marker": ["runs-adapters/runs"],
|
||||
"hello-complete": ["runs-adapters/runs"],
|
||||
"plan-revise-accept": ["questions-and-approvals/plan-review"],
|
||||
"plan-approve-complete": ["questions-and-approvals/plan-review"],
|
||||
"ask-question": ["questions-and-approvals/questions"],
|
||||
"question-resume-complete": ["questions-and-approvals/questions", "questions-and-approvals/continuation"],
|
||||
});
|
||||
associate(["local-session-integrity", "extended-harnesses", "grok-qualification", "grok-subscription-qualification"], {
|
||||
"structured-question-resume": ["questions-and-approvals/continuation", "runs-adapters/session"],
|
||||
"structured-question-restart-resume": ["questions-and-approvals/continuation", "runs-adapters/session"],
|
||||
});
|
||||
associate(["question-resume"], {
|
||||
"question-answer-resume": ["questions-and-approvals/questions", "questions-and-approvals/continuation"],
|
||||
});
|
||||
associate(["continuation"], {
|
||||
"answer-updates-scope": ["questions-and-approvals/questions", "questions-and-approvals/continuation"],
|
||||
"clarification-not-approval": ["questions-and-approvals/questions", "questions-and-approvals/confirmations"],
|
||||
"revision-preserves-approval": ["questions-and-approvals/plan-review"],
|
||||
"completed-action-resume": ["questions-and-approvals/continuation"],
|
||||
"question-tool-documentation": ["questions-and-approvals/questions"],
|
||||
"provider-question-bridge": ["questions-and-approvals/questions", "questions-and-approvals/continuation"],
|
||||
});
|
||||
associate(["first-task", "confirmation-replies", "completion-updates"], Object.fromEntries([
|
||||
"interview-first-response", "clear-task-first-response", "ambiguous-task-first-response",
|
||||
"plain-message-first-response", "plan-first-response", "interview-plan-accept",
|
||||
"task-card-accept", "accept-while-running", "task-reply-accept", "clarify-propose-accept",
|
||||
"revise-accept", "reject-no-execution",
|
||||
].map((id) => [id, ["onboarding/first-task"]])));
|
||||
associate(["first-task"], { "ordinary-task-control": ["tasks/completion"] });
|
||||
associate(["confirmation-replies"], {
|
||||
"confirmation-ambiguous": ["questions-and-approvals/confirmations", "questions-and-approvals/audience"],
|
||||
"unanswered-question-return": ["questions-and-approvals/dismiss-and-reopen", "questions-and-approvals/continuation"],
|
||||
});
|
||||
associate(["agent-chat", "agent-chat-hardening", "grok-qualification", "grok-subscription-qualification", "stock-harness"], {
|
||||
"create-backlog": ["tasks/create"],
|
||||
"reassign-task": ["tasks/properties"],
|
||||
"continuity-restart": ["agent-chat/history", "runs-adapters/session"],
|
||||
"new-session": ["agent-chat/history", "runs-adapters/session"],
|
||||
"stop-new-resume": ["steering/stop-and-pause", "runs-adapters/session"],
|
||||
"stop-startup-new-resume": ["steering/stop-and-pause", "runs-adapters/session"],
|
||||
"plan-handoff": ["agent-chat/handoff", "questions-and-approvals/plan-review"],
|
||||
"clarify-reuse": ["projects/task-context"],
|
||||
"multi-repository": ["projects/repositories"],
|
||||
"hire-delegate-reuse": ["agents/hire", "task-coordination/delegation"],
|
||||
"committed-send-retry": ["recovery/unknown-outcome"],
|
||||
});
|
||||
associate(["agent-chat-qualification"], {
|
||||
"active-reassignment": ["tasks/properties"],
|
||||
"worker-crash-retry": ["recovery/source-action"],
|
||||
});
|
||||
associate(["agent-chat-stories"], {
|
||||
"enable-disable-resume": ["agent-chat/history"],
|
||||
"followup-while-running": ["steering/queue"],
|
||||
"revise-while-running": ["steering/steer"],
|
||||
});
|
||||
associate(["completion-updates"], Object.fromEntries([
|
||||
"handoff-completion-idle", "handoff-completion-busy", "handoff-completion-multiple", "handoff-completion-restart",
|
||||
].map((id) => [id, ["agent-chat/handoff", "tasks/completion"]])));
|
||||
associate(["everyday-workflows", "native-connection-guidance", "grok-qualification", "grok-subscription-qualification"], {
|
||||
"build-revise": ["documents-artifacts/work-products"],
|
||||
"delegate-feedback": ["task-coordination/delegation", "steering/steer"],
|
||||
"agent-review-handoff": ["task-coordination/signoff"],
|
||||
"hire-reuse": ["agents/hire", "task-coordination/delegation"],
|
||||
"service-approve": ["questions-and-approvals/tool-review", "app-permissions/action-policy"],
|
||||
"service-decline": ["questions-and-approvals/tool-review", "app-permissions/action-policy"],
|
||||
"connection-decline": ["connection-setup/task-request"],
|
||||
"provider-decline": ["connection-setup/task-request"],
|
||||
"provider-second": ["connection-setup/task-request"],
|
||||
"provider-native": ["connection-setup/task-request"],
|
||||
"agentmail-setup": ["connection-setup/task-request"],
|
||||
"recover-controller": ["recovery/user-continuation", "documents-artifacts/work-products"],
|
||||
"stop-redirect": ["steering/stop-and-pause"],
|
||||
"create-skill-studio": ["skills/authoring"],
|
||||
});
|
||||
associate(["hiring-templates"], { "hire-coder-template-reuse": ["agents/hire", "task-coordination/delegation"] });
|
||||
associate(["task-workspaces"], {
|
||||
"task-directory-repository-resume": ["workspaces/binding", "workspaces/provisioning", "projects/repositories"],
|
||||
});
|
||||
associate(["task-titles"], {
|
||||
"prompt-title-standard": ["tasks/create"],
|
||||
"prompt-title-ask": ["tasks/create"],
|
||||
"preserve-explicit-title": ["tasks/create"],
|
||||
});
|
||||
associate(["pi-controls", "native-active-stop"], {
|
||||
"pending-permission-stop": ["steering/stop-and-pause"],
|
||||
"same-turn-steering": ["steering/steer"],
|
||||
});
|
||||
|
||||
export function journeyCapabilities(execution: Pick<ReportExecution, "suite" | "task">) {
|
||||
return (associations.get(`${execution.suite.id}/${execution.task.id}`) ?? [])
|
||||
.map((id) => capabilities.get(id)!);
|
||||
}
|
||||
|
||||
export function journeyFeatureIds(execution: Pick<ReportExecution, "suite" | "task">) {
|
||||
return [...new Set(journeyCapabilities(execution).map((capability) => capability.feature.id))];
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
import { once } from "node:events";
|
||||
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import type { Server } from "node:http";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { createRunnerReportPreview } from "./preview.js";
|
||||
|
||||
const directories: string[] = [];
|
||||
const servers: Server[] = [];
|
||||
const campaignId = "preview-test";
|
||||
const campaignRoot = `campaigns/${campaignId}`;
|
||||
const screenshotPath = `${campaignRoot}/evidence/screenshot.png`;
|
||||
const missingPath = `${campaignRoot}/evidence/missing.png`;
|
||||
const screenshot = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Wl6nAAAAABJRU5ErkJggg==", "base64");
|
||||
const files = new Map([
|
||||
["history.json", Buffer.from(JSON.stringify({
|
||||
schema: "paperclip.runner-e2e.history/v1", campaigns: [{ campaignId }],
|
||||
}))],
|
||||
[`${campaignRoot}/bundle-manifest.json`, Buffer.from(JSON.stringify({
|
||||
campaignId,
|
||||
files: ["evidence/screenshot.png", "evidence/missing.png"].map((file) => ({ path: file, bytes: screenshot.length, sha256: "fixture" })),
|
||||
}))],
|
||||
[screenshotPath, screenshot],
|
||||
]);
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(servers.splice(0).map((server) => new Promise<void>((resolve, reject) => {
|
||||
server.close((error) => error ? reject(error) : resolve());
|
||||
server.closeAllConnections();
|
||||
})));
|
||||
await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true })));
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
async function cacheDirectory() {
|
||||
const directory = await mkdtemp(path.join(os.tmpdir(), "runner-report-preview-"));
|
||||
directories.push(directory);
|
||||
return path.join(directory, "cache");
|
||||
}
|
||||
|
||||
async function listen(options: Parameters<typeof createRunnerReportPreview>[0]) {
|
||||
const { server } = await createRunnerReportPreview(options);
|
||||
servers.push(server);
|
||||
server.listen(0, "127.0.0.1");
|
||||
await once(server, "listening");
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") throw new Error("No preview port");
|
||||
return `http://127.0.0.1:${address.port}/runner-e2e/`;
|
||||
}
|
||||
|
||||
describe("local report preview boundaries", () => {
|
||||
it("serves and caches listed evidence while rejecting unlisted and encoded traversal paths", async () => {
|
||||
const cacheRoot = await cacheDirectory();
|
||||
const privateBytes = Buffer.from("outside the public cache");
|
||||
await writeFile(path.join(cacheRoot, "../private.png"), privateBytes);
|
||||
const download = vi.fn(async (relative: string) => {
|
||||
const bytes = files.get(relative);
|
||||
if (!bytes) throw new Error(`Unexpected download: ${relative}`);
|
||||
return bytes;
|
||||
});
|
||||
const origin = await listen({ cacheRoot, download });
|
||||
const response = await fetch(`${origin}${screenshotPath}`);
|
||||
expect(response.status).toBe(200);
|
||||
expect(response.headers.get("content-type")).toBe("image/png");
|
||||
expect(Buffer.from(await response.arrayBuffer())).toEqual(screenshot);
|
||||
expect(await readFile(path.join(cacheRoot, screenshotPath))).toEqual(screenshot);
|
||||
await fetch(`${origin}${screenshotPath}`).then((response) => response.arrayBuffer());
|
||||
expect(download.mock.calls.filter(([relative]) => relative === screenshotPath)).toHaveLength(1);
|
||||
await writeFile(path.join(cacheRoot, campaignRoot, "private.png"), privateBytes);
|
||||
|
||||
vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
for (const relative of [
|
||||
`${campaignRoot}/private.png`,
|
||||
`${campaignRoot}/%2e%2e%2f%2e%2e%2f%2e%2e%2fprivate.png`,
|
||||
`${campaignRoot}/evidence/%2fprivate.png`,
|
||||
]) {
|
||||
const blocked = await fetch(`${origin}${relative}`);
|
||||
expect(blocked.status).not.toBe(200);
|
||||
expect(await blocked.text()).not.toContain(privateBytes.toString());
|
||||
}
|
||||
expect(download.mock.calls.map(([relative]) => relative)).toEqual([
|
||||
"history.json", `${campaignRoot}/bundle-manifest.json`, screenshotPath,
|
||||
]);
|
||||
});
|
||||
|
||||
it("serves cached evidence offline and never downloads a missing listed file", async () => {
|
||||
const cacheRoot = await cacheDirectory();
|
||||
for (const [relative, bytes] of files) {
|
||||
const filename = path.join(cacheRoot, relative);
|
||||
await mkdir(path.dirname(filename), { recursive: true });
|
||||
await writeFile(filename, bytes);
|
||||
}
|
||||
const download = vi.fn(async () => { throw new Error("Offline previews must not download"); });
|
||||
const origin = await listen({ cacheRoot, offline: true, refresh: true, download });
|
||||
const cached = await fetch(`${origin}${screenshotPath}`);
|
||||
expect(cached.status).toBe(200);
|
||||
expect(Buffer.from(await cached.arrayBuffer())).toEqual(screenshot);
|
||||
vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
const missing = await fetch(`${origin}${missingPath}`);
|
||||
expect(missing.status).toBe(502);
|
||||
expect(await missing.text()).toContain(`Not cached: ${missingPath}`);
|
||||
expect(download).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,222 @@
|
||||
import { createServer } from "node:http";
|
||||
import { mkdir, readFile, rename, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { renderRunnerE2EDashboard } from "./dashboard.js";
|
||||
import { renderRunnerHistoryIndex } from "./history-index.js";
|
||||
import type { RunnerE2ECampaign, RunnerE2EHistoryIndex, RunnerE2EResult } from "./types.js";
|
||||
|
||||
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
|
||||
const publicRoot = "https://d1p6rlowie26tp.cloudfront.net/runner-e2e/";
|
||||
const routeRoot = "/runner-e2e/";
|
||||
export async function createRunnerReportPreview(options: {
|
||||
cacheRoot?: string;
|
||||
offline?: boolean;
|
||||
refresh?: boolean;
|
||||
download?: (relative: string) => Promise<Buffer>;
|
||||
} = {}) {
|
||||
const cacheRoot = options.cacheRoot ?? path.join(repositoryRoot, "tests/runner-e2e/results/design-preview");
|
||||
const offline = options.offline ?? false;
|
||||
const downloadFile = options.download ?? (async (relative: string) => {
|
||||
const response = await fetch(new URL(relative, publicRoot), {
|
||||
signal: AbortSignal.timeout(60_000),
|
||||
redirect: "error",
|
||||
});
|
||||
if (!response.ok) throw new Error(`Public artifact ${relative}: HTTP ${response.status}`);
|
||||
return Buffer.from(await response.arrayBuffer());
|
||||
});
|
||||
const version = String(Date.now());
|
||||
const inFlight = new Map<string, Promise<Buffer>>();
|
||||
|
||||
interface Manifest {
|
||||
campaignId: string;
|
||||
files: Array<{ path: string; sha256: string; bytes: number }>;
|
||||
}
|
||||
type PublishedResult = RunnerE2EResult & { evidenceValid?: boolean; evidenceErrors?: string[] };
|
||||
type PublishedCampaign = Omit<RunnerE2ECampaign, "results"> & { results: PublishedResult[] };
|
||||
|
||||
function safePath(relative: string) {
|
||||
if (!relative || relative.split("/").some((part) =>
|
||||
!part || part === "." || part === ".." || !/^[A-Za-z0-9._-]+$/.test(part),
|
||||
)) throw new Error("Invalid public artifact path");
|
||||
return relative;
|
||||
}
|
||||
|
||||
async function publicFile(relative: string, refresh = false): Promise<Buffer> {
|
||||
safePath(relative);
|
||||
const filename = path.join(cacheRoot, relative);
|
||||
if (!refresh || offline) {
|
||||
try { return await readFile(filename); }
|
||||
catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; }
|
||||
}
|
||||
if (offline) throw new Error(`Not cached: ${relative}. Run once without --offline.`);
|
||||
const pending = inFlight.get(relative);
|
||||
if (pending) return pending;
|
||||
const download = (async () => {
|
||||
const bytes = await downloadFile(relative);
|
||||
await mkdir(path.dirname(filename), { recursive: true });
|
||||
const temporary = `${filename}.${process.pid}.tmp`;
|
||||
await writeFile(temporary, bytes);
|
||||
await rename(temporary, filename);
|
||||
return bytes;
|
||||
})();
|
||||
inFlight.set(relative, download);
|
||||
try { return await download; }
|
||||
finally { inFlight.delete(relative); }
|
||||
}
|
||||
|
||||
async function publicJson<T>(relative: string, refresh = false): Promise<T> {
|
||||
return JSON.parse((await publicFile(relative, refresh)).toString("utf8")) as T;
|
||||
}
|
||||
|
||||
const retainedHistory = await publicJson<RunnerE2EHistoryIndex>("history.json", options.refresh ?? false);
|
||||
if (retainedHistory.schema !== "paperclip.runner-e2e.history/v1") throw new Error("Unsupported public history");
|
||||
const campaigns = new Set(retainedHistory.campaigns.map((campaign) => campaign.campaignId));
|
||||
const localHistory: RunnerE2EHistoryIndex = {
|
||||
...retainedHistory,
|
||||
campaigns: retainedHistory.campaigns.map((campaign) => ({
|
||||
...campaign,
|
||||
publicUrl: `${routeRoot}campaigns/${encodeURIComponent(campaign.campaignId)}/`,
|
||||
})),
|
||||
};
|
||||
const manifests = new Map<string, Promise<Manifest>>();
|
||||
function manifestFor(campaignId: string) {
|
||||
if (!campaigns.has(campaignId)) throw new Error("Unknown campaign");
|
||||
let pending = manifests.get(campaignId);
|
||||
if (!pending) {
|
||||
pending = publicJson<Manifest>(`campaigns/${campaignId}/bundle-manifest.json`).then((manifest) => {
|
||||
if (manifest.campaignId !== campaignId || !Array.isArray(manifest.files)) throw new Error("Invalid campaign manifest");
|
||||
for (const file of manifest.files) safePath(file.path);
|
||||
return manifest;
|
||||
});
|
||||
manifests.set(campaignId, pending);
|
||||
pending.catch(() => manifests.delete(campaignId));
|
||||
}
|
||||
return pending;
|
||||
}
|
||||
|
||||
async function campaignHtml(campaignId: string) {
|
||||
const [campaign, manifest] = await Promise.all([
|
||||
publicJson<PublishedCampaign>(`campaigns/${campaignId}/normalized-results.json`),
|
||||
manifestFor(campaignId),
|
||||
]);
|
||||
if (campaign.campaignId !== campaignId || !Array.isArray(campaign.results)) throw new Error("Invalid retained campaign");
|
||||
return renderRunnerE2EDashboard({
|
||||
title: "Runner Full-Stack E2E",
|
||||
generatedAt: campaign.generatedAt,
|
||||
expected: campaign.expected,
|
||||
catalog: runnerMatrix,
|
||||
campaign,
|
||||
history: localHistory,
|
||||
entries: campaign.results.map((result) => {
|
||||
const base = `evidence/${result.executionId}/attempt-${result.attempt}`;
|
||||
return {
|
||||
result,
|
||||
valid: result.status === "passed" && result.cleanup === "passed" && result.evidenceValid !== false,
|
||||
errors: result.evidenceErrors ?? [],
|
||||
evidenceBaseHref: base,
|
||||
evidenceFiles: manifest.files.filter((file) => file.path.startsWith(`${base}/`))
|
||||
.map((file) => file.path.slice(base.length + 1)),
|
||||
};
|
||||
}),
|
||||
});
|
||||
}
|
||||
|
||||
function withReload(html: string) {
|
||||
return html.replace("</body>", `<script>
|
||||
const previewVersion = ${JSON.stringify(version)};
|
||||
setInterval(async () => {
|
||||
try {
|
||||
const response = await fetch("/__preview/version", { cache: "no-store" });
|
||||
if (response.ok && await response.text() !== previewVersion) location.reload();
|
||||
} catch {}
|
||||
}, 1000);
|
||||
</script></body>`);
|
||||
}
|
||||
|
||||
const contentTypes: Record<string, string> = {
|
||||
".json": "application/json", ".png": "image/png", ".woff2": "font/woff2",
|
||||
".txt": "text/plain; charset=utf-8", ".md": "text/plain; charset=utf-8",
|
||||
".log": "text/plain; charset=utf-8", ".xml": "application/xml",
|
||||
};
|
||||
const server = createServer(async (request, response) => {
|
||||
response.setHeader("Cache-Control", "no-store");
|
||||
try {
|
||||
if (request.method !== "GET" && request.method !== "HEAD") {
|
||||
response.writeHead(405).end("Read-only preview");
|
||||
return;
|
||||
}
|
||||
const pathname = new URL(request.url ?? "/", "http://localhost").pathname;
|
||||
if (pathname === "/__preview/version") {
|
||||
response.writeHead(200, { "Content-Type": "text/plain" }).end(version);
|
||||
return;
|
||||
}
|
||||
if (pathname === "/" || pathname === "/runner-e2e") {
|
||||
response.writeHead(302, { Location: routeRoot }).end();
|
||||
return;
|
||||
}
|
||||
let html: string | undefined;
|
||||
let bytes: Buffer | undefined;
|
||||
let extension = ".html";
|
||||
if (pathname === routeRoot || pathname === `${routeRoot}index.html`) {
|
||||
html = renderRunnerHistoryIndex(localHistory);
|
||||
} else if (pathname.startsWith(routeRoot)) {
|
||||
const relative = safePath(decodeURIComponent(pathname.slice(routeRoot.length)).replace(/\/$/, ""));
|
||||
const campaignMatch = /^campaigns\/([A-Za-z0-9._-]+)(?:\/(.*))?$/.exec(relative);
|
||||
if (campaignMatch && campaigns.has(campaignMatch[1]!)) {
|
||||
const campaignId = campaignMatch[1]!;
|
||||
const artifact = campaignMatch[2];
|
||||
if (!artifact || artifact === "index.html" || artifact === "dashboard.html") {
|
||||
if (!artifact && !pathname.endsWith("/")) {
|
||||
response.writeHead(302, { Location: `${pathname}/` }).end();
|
||||
return;
|
||||
}
|
||||
html = await campaignHtml(campaignId);
|
||||
} else if ((await manifestFor(campaignId)).files.some((file) => file.path === artifact)) {
|
||||
extension = path.extname(artifact);
|
||||
if (contentTypes[extension]) bytes = await publicFile(relative);
|
||||
}
|
||||
} else if (relative === "history.json") {
|
||||
extension = ".json";
|
||||
bytes = Buffer.from(JSON.stringify(retainedHistory));
|
||||
} else if (relative === "assets/InterVariable.woff2" || relative === "assets/favicon-32x32.png") {
|
||||
extension = path.extname(relative);
|
||||
bytes = await publicFile(relative);
|
||||
}
|
||||
}
|
||||
if (html !== undefined) bytes = Buffer.from(withReload(html));
|
||||
if (!bytes) {
|
||||
response.writeHead(404, { "Content-Type": "text/plain" }).end("Preview page or public artifact not found");
|
||||
return;
|
||||
}
|
||||
response.writeHead(200, { "Content-Type": contentTypes[extension] ?? "text/html; charset=utf-8" });
|
||||
response.end(request.method === "HEAD" ? undefined : bytes);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
console.error(message);
|
||||
response.writeHead(502, { "Content-Type": "text/plain; charset=utf-8" }).end(message);
|
||||
}
|
||||
});
|
||||
return { server, campaignCount: campaigns.size, cacheRoot };
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = process.argv.slice(2).filter((argument) => argument !== "--");
|
||||
const portIndex = args.indexOf("--port");
|
||||
const port = portIndex < 0 ? 4317 : Number(args[portIndex + 1]);
|
||||
if (!Number.isInteger(port) || port < 0 || port > 65535) throw new Error("Invalid --port");
|
||||
const { server, campaignCount, cacheRoot } = await createRunnerReportPreview({
|
||||
offline: args.includes("--offline"), refresh: args.includes("--refresh"),
|
||||
});
|
||||
server.listen(port, "127.0.0.1", () => {
|
||||
const address = server.address();
|
||||
const origin = `http://127.0.0.1:${typeof address === "object" && address ? address.port : port}`;
|
||||
console.log(`History: ${origin}${routeRoot}`);
|
||||
console.log(`${campaignCount} real campaigns; reports and public evidence cache on first access.`);
|
||||
console.log(`Cached data: ${cacheRoot}`);
|
||||
console.log("Edit history-index.ts or dashboard.ts; Node restarts the preview and the browser reloads.");
|
||||
});
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) await main();
|
||||
@@ -10,6 +10,7 @@ import { discoverReportCatalog } from "./report-catalog.js";
|
||||
import { buildRunnerCampaign } from "./history.js";
|
||||
import { renderRunnerE2EDashboard } from "./dashboard.js";
|
||||
import { regenerateRunnerDashboard } from "./dashboard-regenerate.js";
|
||||
import { journeyCapabilities } from "./journey-features.js";
|
||||
import type { RunnerE2EResult } from "./types.js";
|
||||
|
||||
const temporary: string[] = [];
|
||||
@@ -58,6 +59,34 @@ function campaign(results: RunnerE2EResult[] = [result()]) {
|
||||
}
|
||||
|
||||
describe("trusted report catalog discovery", () => {
|
||||
it("shows only the campaign selection while retaining a missing selected result", () => {
|
||||
const known = runnerMatrix.find((execution) => execution.id === "core-compatibility.legacy-codex.local.ask-question")!;
|
||||
const missing = runnerMatrix.find((execution) => execution.id === "core-compatibility.legacy-codex.local.message-marker")!;
|
||||
const page = renderRunnerE2EDashboard({
|
||||
title: "Narrow campaign", generatedAt: "2026-10-20T12:00:00Z",
|
||||
expected: [known.id, missing.id], catalog: runnerMatrix,
|
||||
entries: [{ result: { ...result(), executionId: known.id, suiteId: known.suite.id,
|
||||
profileId: known.profile.id, environmentId: known.environment.id, caseId: known.task.id }, valid: true, errors: [] }],
|
||||
});
|
||||
expect(page).toContain(`data-execution-id="${known.id}"`);
|
||||
expect(page).toContain(`data-execution-id="${missing.id}"`);
|
||||
expect(page).toContain('class="case case-missing"');
|
||||
expect(page).not.toContain('id="suite-first-task"');
|
||||
expect(page).not.toContain('data-report-profile="runner-copilot"');
|
||||
expect(page).not.toContain('class="suite-nav"');
|
||||
expect(page.match(/data-report-case\s/g)).toHaveLength(2);
|
||||
expect(page).toContain('data-report-features="questions-and-approvals"');
|
||||
expect(page).toContain('title="questions-and-approvals/questions"');
|
||||
expect(page).toContain("No mapped journey");
|
||||
expect(page).toContain("0/1 passed · 1 missing");
|
||||
});
|
||||
|
||||
it("leaves an unfamiliar journey unmapped even when its case name matches", () => {
|
||||
const known = runnerMatrix.find((execution) => execution.id === "core-compatibility.legacy-codex.local.ask-question")!;
|
||||
expect(journeyCapabilities(known).map((capability) => capability.id)).toEqual(["questions-and-approvals/questions"]);
|
||||
expect(journeyCapabilities({ ...known, suite: { ...known.suite, id: "branch-only" } })).toEqual([]);
|
||||
});
|
||||
|
||||
it("aggregates unknown suites and renders their cases, missing results, and screenshot gallery", () => {
|
||||
const summary = campaign();
|
||||
expect(summary.suites).toEqual([
|
||||
|
||||
@@ -319,10 +319,10 @@ describe("runner E2E report aggregation", () => {
|
||||
expect(dashboard).toContain("Runner Full-Stack E2E");
|
||||
expect(dashboard).toContain(executionId);
|
||||
expect(dashboard).toContain("case-passed");
|
||||
expect(dashboard).toContain(
|
||||
expect(dashboard).not.toContain(
|
||||
"core-compatibility.runner-acpx-codex.daytona.message-marker",
|
||||
);
|
||||
expect(dashboard).toContain("case-not-selected");
|
||||
expect(dashboard).not.toContain('class="case case-not-selected"');
|
||||
expect(dashboard).toContain("<img");
|
||||
expect(dashboard).toContain('class="brand-lockup"');
|
||||
expect(dashboard).toContain("data-gallery-dialog");
|
||||
@@ -339,7 +339,7 @@ describe("runner E2E report aggregation", () => {
|
||||
"Declared screenshots and normalized results published",
|
||||
);
|
||||
expect(dashboard).toContain("message_contains");
|
||||
expect(dashboard).toContain("Matchers and test context");
|
||||
expect(dashboard).toContain("Test details");
|
||||
expect(dashboard).toContain("Scheduler");
|
||||
expect(dashboard).toContain("Run duration");
|
||||
expect(dashboard).toContain("100ms");
|
||||
@@ -361,9 +361,9 @@ describe("runner E2E report aggregation", () => {
|
||||
expect(dashboard).toContain("data-report-profile");
|
||||
expect(dashboard).toContain("data-report-environment");
|
||||
expect(dashboard).toContain("data-report-status");
|
||||
expect(dashboard.indexOf('class="report-filters"')).toBeGreaterThan(
|
||||
dashboard.indexOf('class="suite-nav"'),
|
||||
);
|
||||
expect(dashboard).not.toContain('class="suite-nav"');
|
||||
expect(dashboard).toContain("data-report-feature");
|
||||
expect(dashboard).toContain("Feature coverage");
|
||||
expect(dashboard).not.toContain(".report-filters { position: sticky");
|
||||
expect(dashboard).toContain("table-layout: fixed");
|
||||
expect(dashboard).toContain('class="profile-column"');
|
||||
|
||||
Reference in new issue
Block a user