mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 20:21:50 +01:00
feat(evals): browse E2E suites across targets and models (#18027)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - E2E tests show whether people can complete real app and agent journeys. > - Operators need to choose a suite and inspect its evidence across deployments and models. > - The history page listed campaigns without a suite browser or clear coverage filters. > - QA smoke also appeared separate from the other retained E2E measurements. > - This pull request puts suites, deployment targets, and named selections in one history browser. > - The benefit is that a passing subset stays distinct from a full suite run, with direct links to evidence. ## Linked Issues or Issue Description Related: #18009 added clear per-test details and links to the tested application. This PR extends the shared E2E history browser. **What happened?** The public directory called full app testing “Product E2E Evals” and made users choose campaigns before they could find a suite. The history list could not filter independently by tested app, agent execution, model, or run selection. A recent passing subset could be mistaken for a full suite result. **Expected behavior** Browse current and retained suites in one E2E category. Keep QA smoke in normal campaign history. Show the latest matching run and latest full suite run. Preserve the recorded definition and denominator when filters change. **Steps to reproduce** 1. Open the E2E history directory. 2. Find a suite, then compare a narrow passing run with a full run across execution environments and models. 3. Find the QA smoke measurement and open its retained screenshots. **Paperclip version or commit** The previous report UI was at `8279c99008`. This branch is rebased on current master. **Deployment mode** Static E2E history and campaign reports generated from retained measurements. ## What Changed - Call full-product testing “E2E tests.” Keep Runner protocol evals separate because they use a mock control plane. - Add a searchable suite sidebar that includes current suites and suites found only in retained history. - Add correlated suite, coverage, deployment target, execution, provider, model, profile, runtime, result, date, and named-selection filters. Keep filters in the URL for reload and back navigation. Sidebar suite links clear other filters; the Suite dropdown keeps deliberate intersections. Incompatible suite/preset URLs explain the empty selection and link to the suite runs. - Show latest matching and latest full run links. Distinguish full, subset, unfinished, and unknown coverage from the recorded suite definition and execution counts. Keep failed full runs visible. - Retain the tested application URL and runtime mode in new history records. Older records show “Not recorded.” Browser-only journeys do not gain an inferred agent or model. - Add an index-only refresh command that can import a reviewed public campaign once. It rejects private Cloud bundles and does not replace retained campaigns or upload evidence. - Keep existing report URLs, test IDs, commands, and measurement timestamps stable. Campaign links carry suite/profile/environment; target/model/provider/runtime remain history-level filters. - Add unit and synthetic browser regressions, including mobile navigation, long source refs, sidebar filter reset, and empty-selection repair. Inset dropdown carets in the shared report styles. Document direct Node entry points without adding package.json shortcuts. - Add a prominent Reset all filters button, blue active-filter borders and tint, and visible active counts for the main and advanced filters. Reset clears all 13 history filters, sidebar search, URL state, and the attention view. Campaign-page filters use the same active styling. ## Verification - Post-rebase focused Vitest: 127 tests passed across 10 files. The latest follow-up selection passed 100 tests across 9 files. E2E support TypeScript passed after both changes. - The synthetic Chromium journey covers full and subset coverage, unfinished selections, correlated filters, browser-only QA selection, URL/back/reload, and desktop and touch navigation. The expanded journey verifies 13 active history fields, the collapsed advanced-filter count, reset of every field and sidebar search, and active border color changes that revert after reset. It makes zero provider calls. - Hub Python tests: 7 passed. The browser harness also exits promptly when Chromium is unavailable; its HTTP server closes on launch failure. - Real history verification covered 419 campaigns and 59 suites. All 418 previous campaign records remained deeply equal after importing the reviewed QA campaign. Publication used the authoritative history ETag. Only mutable index/history/pointer files were written; immutable campaign bundles stayed unchanged. - A live browser walkthrough passed from the two-family eval hub through QA smoke, Basic app journeys, target filtering, the retained report, screenshot gallery, history, and the logo link. The QA URL was clickable and the image loaded. No browser JavaScript errors occurred. The original QA normalized results and all three PNG hashes remained unchanged. - Live [E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history), [core compatibility suite](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history&suite=core-compatibility), and [QA smoke runs](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history&preset=qa-smoke). - The live follow-up verified the incompatible Stock harness/QA-smoke URL, the repair link to 21 Stock harness campaigns, and sidebar navigation that clears preset/model filters. Dropdown carets were inspected in dark, light, and mobile views. No JavaScript errors occurred. - The final live follow-up verified reset of history controls, URL state, sidebar search, and the attention view, plus reload/back restoration and mobile touch reset. Active filter borders and tint were inspected in dark and light views. Campaign-page reset restored all three QA cases. All 419 history records and original QA measurement/evidence hashes stayed unchanged. - Full repository typecheck, test, build, and E2E CI gates passed on final head `1334cb79a5`. Both enforced aggregates, `ci / verify` and `ci / e2e`, completed successfully. Fresh Greptile review scored that exact head 5/5 with no unresolved review threads. ## Risks - Historical metadata can be missing. Unknown targets and runtime details stay explicit instead of being inferred from a worker environment. - Full suite coverage follows the saved definition and selection. Filtering to one model does not redefine that denominator. The latest full-run shortcut can cross coverage selections and says so in its detail text. - This changes shared report presentation. It adds no application runtime or database changes. The browser harness and index refresh do not invoke models. - Public history publication must keep its existing conditional-write and retained-evidence boundaries. Private bundles, raw logs, credentials, and screenshots are not part of this PR. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, code execution, and browser tools. The exact backend model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
b25f3e2af8
commit
d8bd8fe9fb
20 files changed
+675
-77
No files matched your search
@@ -1,5 +1,43 @@
|
||||
# Paperclip evaluation guide
|
||||
|
||||
The public directory calls full-product testing **E2E tests**. Suites describe
|
||||
what is tested; deployment targets describe which app instance was tested.
|
||||
QA smoke is a named selection of existing journeys against QA, and appears in
|
||||
the same campaign history as other E2E runs. Agent execution (local or Daytona),
|
||||
provider, model and profile are independent filters. Browser-only journeys need
|
||||
no agent or model. Historical records without target metadata remain explicitly
|
||||
"Not recorded" rather than being assigned an inferred environment.
|
||||
|
||||
The history sidebar includes the current suite catalog and retained historical
|
||||
suites. Each suite has its latest matching run and latest full run, with full
|
||||
coverage determined from the recorded suite definition, selection and execution
|
||||
counts. An unexecuted full selection remains unfinished, and a subset
|
||||
can pass without qualifying as a full suite run; failed full selections remain
|
||||
visible. Changing a model filter does not redefine the suite's recorded coverage.
|
||||
Existing `runner-e2e` URLs, test IDs, commands, and measurement timestamps remain
|
||||
stable. **Runner protocol evals** remain a separate category because they test
|
||||
the runner/provider against a mock control plane.
|
||||
|
||||
Sidebar suite links open that suite's campaign history with other filters cleared.
|
||||
The Suite dropdown combines with the current filters instead. Explicit URLs retain
|
||||
their selected filters; when a suite is absent from a preset, the empty state
|
||||
explains the combination and links to the suite's runs.
|
||||
Active filters have blue borders and tinted fields. The filter toolbar counts
|
||||
active values, including a count beside More filters for collapsed controls.
|
||||
Reset all filters clears every campaign filter and the sidebar suite search.
|
||||
|
||||
`node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/history-browser.browser.mjs`
|
||||
exercises the history UI using synthetic metadata and Chromium, with no model calls. It checks suite coverage, correlated
|
||||
suite/target/model/execution filters, saved QA selections, URL history and mobile
|
||||
navigation. Install the existing Playwright Chromium dependency before running.
|
||||
For presentation-only refreshes,
|
||||
`node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/history-refresh.ts --history <history.json> --output <directory>`
|
||||
renders an existing history file. Its optional `--public-campaign`, `--public-url` and
|
||||
`--preset` arguments import a reviewed public measurement without replacing
|
||||
existing campaigns. Private Cloud bundles are rejected. The output contains
|
||||
only index HTML and history/pointer JSON; it does not upload evidence or change
|
||||
retained campaign bundles.
|
||||
|
||||
The explicit-only [chat feature evals](../tests/runner-e2e/CHAT-FEATURES.md) promote
|
||||
two natural-language task scenarios into local native Product E2E cells. They
|
||||
verify saving an unstarted task and delegating a saved announcement back to chat,
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="description" content="Paperclip evaluation results, run histories, and guides for Runner Evals and Product E2E Evals.">
|
||||
<meta name="description" content="Paperclip E2E tests and Runner protocol evals: suites, deployment targets, results, screenshots, and run history.">
|
||||
<title>Paperclip Evals</title>
|
||||
<style>
|
||||
:root { color-scheme: light dark; --bg:#f8f8f5; --card:#fff; --text:#20221e; --muted:#60645c; --line:#d9ddd3; --accent:#306340; --tint:#edf3ec; }
|
||||
@@ -49,23 +49,23 @@
|
||||
<p class="intro">Two ways to test Paperclip. Follow each program from a test case to its evidence, results, and run history.</p>
|
||||
<div class="grid">
|
||||
<article>
|
||||
<div class="number">01 / RUNNER</div>
|
||||
<h2>Runner Evals</h2>
|
||||
<div class="number">01 / E2E</div>
|
||||
<h2>E2E tests</h2>
|
||||
<p>Can a user complete work through Paperclip, from the first request to a usable result?</p>
|
||||
<div class="boundary">Real app journeys, including browser-only and agent workflows<br>Filter by suite, deployment target, agent execution, and model</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$product_counts</div><div class="meta">$product_date · $product_coverage</div><div class="id">$product_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history">Browse suites and runs →</a><a href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history&preset=qa-smoke">QA smoke runs</a><a href="$product_url">Latest report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/tests/runner-e2e/README.md">E2E operations guide</a>
|
||||
</article>
|
||||
<article>
|
||||
<div class="number">02 / RUNNER PROTOCOL</div>
|
||||
<h2>Runner protocol evals</h2>
|
||||
<p>Does the runner use tools correctly, respect permissions, and preserve its execution contract?</p>
|
||||
<div class="boundary">Real runner + model provider<br>Controlled mock Paperclip control plane</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$runner_counts</div><div class="meta">$runner_date · $runner_coverage</div><div class="id">$runner_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html">Browse run history →</a><a href="$runner_url">Open report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/packages/paperclip-runner/docs/runner-protocol-live-evals.md">Runner operations guide</a>
|
||||
</article>
|
||||
<article>
|
||||
<div class="number">02 / PRODUCT</div>
|
||||
<h2>Product E2E Evals</h2>
|
||||
<p>Can a user complete work through Paperclip, from the first request to a usable result?</p>
|
||||
<div class="boundary">Real browser + server + database + runner + model provider<br>Local and Daytona environments, as selected</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$product_counts</div><div class="meta">$product_date · $product_coverage</div><div class="id">$product_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/">Browse run history →</a><a href="$product_url">Open report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/tests/runner-e2e/README.md">Product E2E operations guide</a>
|
||||
</article>
|
||||
</div>
|
||||
<p class="note">Snapshot refreshed $refreshed. History pages may contain newer runs. A partial campaign covers only its selected cases. These programs measure different boundaries; their pass rates are not combined.</p>
|
||||
<section class="start"><div><h2>Choose a test. Make the result inspectable.</h2><p>The eval guide explains which program to use, how to add a case, and how to distinguish a product defect from model behavior, a grading error, or a setup failure. Evalbook is a report framework used to present evidence; it is not an execution boundary.</p></div><a class="button" href="$guide_url">Read the eval guide →</a></section>
|
||||
|
||||
@@ -182,7 +182,7 @@ export async function regenerateRunnerDashboard(input: {
|
||||
}),
|
||||
);
|
||||
const dashboard = renderRunnerE2EDashboard({
|
||||
title: normalized.presentation?.title ?? "Runner Full-Stack E2E",
|
||||
title: normalized.presentation?.title ?? "E2E tests",
|
||||
generatedAt: normalized.generatedAt,
|
||||
expected,
|
||||
catalog: runnerMatrix,
|
||||
|
||||
@@ -868,9 +868,9 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
<section data-campaign-overview>
|
||||
<header class="report-header">
|
||||
<div>
|
||||
<p class="eyebrow">Full-stack acceptance campaign</p>
|
||||
<p class="eyebrow">E2E campaign</p>
|
||||
<h1 tabindex="-1">${html(input.presentation?.title ?? input.title)}</h1>
|
||||
<p class="lede">${html(input.presentation?.description ?? "Results across runner profiles and execution environments. Declared PNG screenshots and normalized results are retained with every published campaign; additional diagnostic evidence remains in the access-controlled workflow artifact.")}</p>
|
||||
<p class="lede">${html(input.presentation?.description ?? "App and agent journeys across deployment targets, profiles, and execution environments. Declared PNG screenshots and normalized results are retained with every published campaign; additional diagnostic evidence remains in the access-controlled workflow artifact.")}</p>
|
||||
${input.campaign ? `<p class="report-source"><span>${html(input.generatedAt)}</span>${input.campaign.source.sha ? `<code>${html(input.campaign.source.sha.slice(0,8))}</code>` : ""}${input.campaign.source.workflowRunUrl ? `<a href="${html(input.campaign.source.workflowRunUrl)}">Source run ↗</a>` : ""}</p>` : ""}
|
||||
</div>
|
||||
<div class="report-actions">
|
||||
@@ -909,7 +909,7 @@ export function renderRunnerE2EDashboard(input: RunnerDashboardInput) {
|
||||
<div data-feature-coverage>${renderJourneyFeatureCoverage(catalog, entryById)}</div>
|
||||
<section class="report-filters" aria-label="Filter test results">
|
||||
<label class="report-search"><span>Search tests</span><input type="search" placeholder="Test, model, provider, execution ID…" autocomplete="off" data-report-query></label>
|
||||
<button class="filter-reset" type="button" data-report-reset>Reset</button>
|
||||
<button class="filter-reset" type="button" data-report-reset>Reset all filters</button>
|
||||
<label class="report-feature"><span>Feature</span><select data-report-feature><option value="">All features</option>${renderJourneyFeatureOptions(catalog)}</select></label>
|
||||
<label><span>Agent profile</span><select data-report-profile><option value="">All profiles</option>${filterProfiles.map((profile) => `<option value="${html(profile.id)}">${html(profile.label)}</option>`).join("")}</select></label>
|
||||
<label><span>Environment</span><select data-report-environment><option value="">All environments</option>${filterEnvironments.map((environment) => `<option value="${html(environment.id)}">${html(environment.label)}</option>`).join("")}</select></label>
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
// Explicit browser regression; synthetic history only, with no Paperclip/model runs.
|
||||
import { createServer } from 'node:http';
|
||||
import { readFile, mkdir } from 'node:fs/promises';
|
||||
import { parseArgs } from 'node:util';
|
||||
import { chromium, expect } from '@playwright/test';
|
||||
import { buildRunnerCampaign, campaignHistoryRecord, emptyRunnerHistory } from './history.ts';
|
||||
import { renderRunnerHistoryIndex } from './history-index.ts';
|
||||
|
||||
const {values} = parseArgs({options:{screenshots:{type:'string'}}});
|
||||
const result = (suite,profile,environment,model,targetUrl,status='passed',attempt=1) => ({
|
||||
schema:'paperclip.runner-e2e.result/v2',executionId:`${suite}.${profile}.${environment}.create`,suiteId:suite,
|
||||
suiteDefinitionHash:'fixture-definition',profileId:profile,environmentId:environment,caseId:'create',provider:profile,
|
||||
model,runtimeMode:'native',targetUrl,attempt,status,cleanup:'passed',durationMs:1000,
|
||||
startedAt:'2026-10-27T12:00:00Z',finishedAt:'2026-10-27T12:00:01Z',
|
||||
});
|
||||
const a=result('fixture-tasks','codex','local','model-a','http://localhost:3100');
|
||||
const b=result('fixture-tasks','claude','daytona','model-b','https://qa.test.paperclip.app','failed');
|
||||
const other=result('fixture-other','codex','daytona','model-a','https://qa.test.paperclip.app');
|
||||
const record=(id,date,results,{full=false,preset}={})=>{
|
||||
const campaign=buildRunnerCampaign({campaignId:id,generatedAt:date,expected:results.map(row=>row.executionId),results});
|
||||
const record=campaignHistoryRecord(campaign,'https://reports.example.test/e2e');
|
||||
record.suites=record.suites.map(suite=>({...suite,expected:suite.suiteId==='fixture-tasks'?2:suite.selected,complete:full}));
|
||||
if(preset)record.preset=preset;
|
||||
return record;
|
||||
};
|
||||
const full=record('full-with-failure','2026-10-25T12:00:00Z',[a,b,other],{full:true});
|
||||
full.source.ref='refs/heads/'+'long-branch'.repeat(20);
|
||||
const subset=record('newer-subset','2026-10-26T12:00:00Z',[a]);
|
||||
const unfinished=record('latest-unfinished','2026-10-27T12:00:00Z',[{...a,status:'failed',attempt:0},{...b,attempt:0}],{full:true});
|
||||
const browserOnly={...result('fixture-app','browser','daytona',null,'https://qa.test.paperclip.app'),provider:null,runtimeMode:'none'};
|
||||
const smoke=record('qa-smoke','2026-10-28T12:00:00Z',[browserOnly],{preset:'qa-smoke'});
|
||||
const history={...emptyRunnerHistory(),updatedAt:'2026-10-28T12:00:00Z',latestCampaignId:'qa-smoke',campaigns:[smoke,unfinished,subset,full]};
|
||||
const html=renderRunnerHistoryIndex(history);
|
||||
const assets=new Map([
|
||||
['/assets/InterVariable.woff2',{bytes:await readFile(new URL('../../ui/public/fonts/InterVariable.woff2',import.meta.url)),type:'font/woff2'}],
|
||||
['/assets/favicon-32x32.png',{bytes:await readFile(new URL('../../ui/public/favicon-32x32.png',import.meta.url)),type:'image/png'}],
|
||||
]);
|
||||
const server=createServer((request,response)=>{
|
||||
const asset=assets.get(new URL(request.url,'http://localhost').pathname);
|
||||
response.setHeader('content-type',asset?.type??'text/html; charset=utf-8');response.end(asset?.bytes??html);
|
||||
});
|
||||
await new Promise(resolve=>server.listen(0,'127.0.0.1',resolve));
|
||||
const base=`http://127.0.0.1:${server.address().port}/`;
|
||||
let browser;
|
||||
try{
|
||||
browser=await chromium.launch();
|
||||
const page=await browser.newPage({viewport:{width:1440,height:1000},colorScheme:'dark'});
|
||||
const errors=[];page.on('pageerror',error=>errors.push(error.message));
|
||||
await page.goto(base+'?view=history');
|
||||
await expect(page.getByRole('navigation',{name:'E2E suites',exact:true})).toBeVisible();
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveCount(4);
|
||||
await page.locator('[data-index-suite-link="fixture-tasks"]').click();
|
||||
await expect(page.getByRole('heading',{name:'fixture tasks',exact:true})).toBeVisible();
|
||||
await expect(page.locator('[data-index-latest="any"] a')).toHaveText('latest-unfinished');
|
||||
await expect(page.locator('[data-index-latest="full"] a')).toHaveText('full-with-failure');
|
||||
await page.getByLabel('Suite coverage',{exact:true}).selectOption('full');
|
||||
await expect(page.locator('[data-index-latest="any"] a')).toHaveText('full-with-failure');
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveCount(1);
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden]) [data-index-row-status]')).toHaveText('failed');
|
||||
await page.getByLabel('Suite coverage',{exact:true}).selectOption('subset');
|
||||
await expect(page.locator('[data-index-latest="any"] a')).toHaveText('newer-subset');
|
||||
await expect(page.locator('[data-index-latest="full"] a')).toHaveText('full-with-failure');
|
||||
await expect(page.locator('[data-index-latest="full"] small')).toContainText('across coverage selections');
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveCount(1);
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveAttribute('data-index-id','newer-subset');
|
||||
await page.reload();await expect(page.getByLabel('Suite coverage',{exact:true})).toHaveValue('subset');
|
||||
await page.goBack();await expect(page.getByLabel('Suite coverage',{exact:true})).toHaveValue('full');
|
||||
await page.getByRole('button',{name:'Reset all filters',exact:true}).click();
|
||||
await page.getByLabel('Suite',{exact:true}).selectOption('fixture-tasks');
|
||||
await page.getByText('More filters',{exact:true}).click();
|
||||
await page.getByLabel('Target',{exact:true}).selectOption('https://qa.test.paperclip.app');
|
||||
await page.getByLabel('Model',{exact:true}).selectOption('model-a');
|
||||
await page.getByLabel('Agent execution',{exact:true}).selectOption('daytona');
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveCount(0);
|
||||
await expect(page.locator('[data-index-empty]')).toBeVisible();
|
||||
await page.getByRole('button',{name:'Reset all filters',exact:true}).click();
|
||||
await page.locator('[data-index-preset-link="qa-smoke"]').click();
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveCount(1);
|
||||
await page.getByLabel('Model',{exact:true}).selectOption('__none__');
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden]) [data-index-row-counts]')).toHaveText('1/1 matching executions passed · 1 executed');
|
||||
await page.locator('[data-index-suite-link="fixture-tasks"]').click();
|
||||
await expect(page.getByLabel('Run preset',{exact:true})).toHaveValue('');
|
||||
await expect(page.getByLabel('Model',{exact:true})).toHaveValue('');
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveCount(3);
|
||||
await page.goto(base+'?view=history&suite=fixture-tasks&preset=qa-smoke');
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveCount(0);
|
||||
await expect(page.locator('[data-index-empty-message]')).toHaveText('No fixture tasks campaigns are included in QA smoke.');
|
||||
await page.getByRole('link',{name:'View fixture tasks runs',exact:true}).click();
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveCount(3);
|
||||
await expect(page.getByLabel('Run preset',{exact:true})).toHaveValue('');
|
||||
await page.locator('[data-index-preset-link="qa-smoke"]').click();
|
||||
await page.getByLabel('Suite',{exact:true}).selectOption('fixture-tasks');
|
||||
await expect(page.getByLabel('Run preset',{exact:true})).toHaveValue('qa-smoke');
|
||||
await expect(page.locator('[data-index-empty-message]')).toHaveText('No fixture tasks campaigns are included in QA smoke.');
|
||||
const selectStyle=await page.getByLabel('Agent profile',{exact:true}).evaluate(select=>{
|
||||
const style=getComputedStyle(select);return {appearance:style.appearance,padding:parseInt(style.paddingRight),arrow:style.backgroundPosition};
|
||||
});
|
||||
if(selectStyle.appearance!=='none'||selectStyle.padding<32||!selectStyle.arrow.includes('12px'))throw Error('Dropdown arrow lacks inset: '+JSON.stringify(selectStyle));
|
||||
await page.getByRole('link',{name:'Overview',exact:true}).click();
|
||||
await expect(page.getByRole('heading',{name:'E2E tests',exact:true})).toBeVisible();
|
||||
await expect(page.getByLabel('Run preset',{exact:true})).toHaveValue('');
|
||||
await expect(page.locator('[data-index-overview]')).toBeVisible();
|
||||
await page.getByLabel('Target',{exact:true}).selectOption('https://qa.test.paperclip.app');
|
||||
await expect(page.locator('[data-index-overview]')).not.toBeVisible();
|
||||
await page.getByRole('button',{name:'Reset all filters',exact:true}).click();
|
||||
const neutralBorder=await page.getByLabel('Suite',{exact:true}).evaluate(control=>getComputedStyle(control).borderColor);
|
||||
const allFilters=new URLSearchParams({view:'attention',suite:'fixture-tasks',coverage:'full',target:'https://qa.test.paperclip.app',
|
||||
environment:'daytona',provider:'claude',model:'model-b',profile:'claude',runtime:'native',preset:'qa-smoke',
|
||||
status:'failed',from:'2026-10-24',through:'2026-10-29',q:'full'});
|
||||
await page.goto(base+'?'+allFilters);
|
||||
await expect(page.locator('[data-index-active-count]')).toHaveText('13 active');
|
||||
await expect(page.locator('[data-index-filter][data-filter-active=true]')).toHaveCount(13);
|
||||
await expect(page.getByLabel('Suite',{exact:true})).not.toHaveCSS('border-color',neutralBorder);
|
||||
await page.getByText('More filters',{exact:true}).click();
|
||||
await expect(page.locator('[data-index-more-count]')).toBeVisible();
|
||||
await expect(page.locator('[data-index-more-count]')).toHaveText('9 active');
|
||||
await page.reload();
|
||||
await expect(page.locator('[data-index-active-count]')).toHaveText('13 active');
|
||||
await page.getByLabel('Find suites',{exact:true}).fill('fixture tasks');
|
||||
await expect(page.locator('[data-index-suite-link]:visible')).toHaveCount(1);
|
||||
await page.getByRole('button',{name:'Reset all filters',exact:true}).click();
|
||||
await expect(page).toHaveURL(base+'?view=history');
|
||||
await expect(page.locator('[data-index-campaign]:not([hidden])')).toHaveCount(4);
|
||||
await expect(page.locator('[data-index-filter][data-filter-active=true]')).toHaveCount(0);
|
||||
await expect(page.locator('[data-index-active-count]')).toHaveText('None active');
|
||||
await expect(page.locator('[data-index-more-count]')).not.toBeVisible();
|
||||
await expect(page.getByLabel('Find suites',{exact:true})).toHaveValue('');
|
||||
await expect(page.getByLabel('Suite',{exact:true})).toHaveCSS('border-color',neutralBorder);
|
||||
await expect(page.locator('[data-index-suite-link]:visible')).toHaveCount(await page.locator('[data-index-suite-link]').count());
|
||||
for(const control of await page.locator('[data-index-filter]').all())await expect(control).toHaveValue('');
|
||||
await page.goBack();
|
||||
await expect(page.locator('[data-index-active-count]')).toHaveText('13 active');
|
||||
await page.getByRole('button',{name:'Reset all filters',exact:true}).click();
|
||||
for(const width of [1440,1000,780,390]){
|
||||
await page.setViewportSize({width,height:1000});
|
||||
if(await page.evaluate(()=>document.documentElement.scrollWidth>innerWidth))throw Error(`horizontal overflow at ${width}`);
|
||||
}
|
||||
const mobile=await browser.newContext({viewport:{width:390,height:844},isMobile:true,hasTouch:true});
|
||||
const phone=await mobile.newPage();await phone.goto(base+'?view=history');
|
||||
await phone.getByRole('button',{name:'Open navigation',exact:true}).tap();
|
||||
await phone.getByLabel('Find suites',{exact:true}).fill('fixture tasks');
|
||||
await phone.locator('[data-index-suite-link="fixture-tasks"]').tap();
|
||||
await expect(phone.locator('.report-sidebar')).not.toBeVisible();
|
||||
await expect(phone.locator('[data-index-latest="full"] a')).toHaveText('full-with-failure');
|
||||
if(values.screenshots){await mkdir(values.screenshots,{recursive:true});await phone.screenshot({path:values.screenshots+'/fixture-mobile.png',fullPage:true});}
|
||||
await mobile.close();
|
||||
if(errors.length)throw Error(errors.join('\n'));
|
||||
console.log('E2E history browser passed: recorded full runs, subsets, unfinished selections, correlated filters, QA preset, URL/back/reload, desktop and touch navigation. No provider calls.');
|
||||
}finally{try{await browser?.close();}finally{await new Promise(resolve=>server.close(resolve));}}
|
||||
@@ -0,0 +1,86 @@
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { buildRunnerCampaign, campaignHistoryRecord, emptyRunnerHistory } from "./history.js";
|
||||
import { fullSuiteSelection, historyBrowserCampaign, historyBrowserJson, historySuiteCatalog, historySuiteCoverage } from "./history-browser.js";
|
||||
import { refreshHistoryIndex } from "./history-refresh.js";
|
||||
import type { RunnerE2EResult } from "./types.js";
|
||||
|
||||
const temporary: string[] = [];
|
||||
afterEach(async () => { await Promise.all(temporary.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
|
||||
function fixture() {
|
||||
const result: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: "app-basics.browser.daytona.create",
|
||||
suiteId: "app-basics", suiteDefinitionHash: "recorded-definition", profileId: "browser", environmentId: "daytona", caseId: "create",
|
||||
provider: null, model: null, runtimeMode: "none", targetUrl: "https://qa.test.paperclip.app/", attempt: 1,
|
||||
status: "passed", cleanup: "passed", startedAt: "2026-10-27T12:00:00Z", finishedAt: "2026-10-27T12:00:01Z", durationMs: 1000 };
|
||||
const campaign = buildRunnerCampaign({ campaignId: "qa-unchanged", generatedAt: result.finishedAt, expected: [result.executionId], results: [result] });
|
||||
return { result, campaign, record: campaignHistoryRecord(campaign, "https://reports.example.test/e2e") };
|
||||
}
|
||||
|
||||
describe("E2E history organization", () => {
|
||||
it("includes unrun catalog suites and historical suites in one catalog", () => {
|
||||
const history = { ...emptyRunnerHistory(), campaigns: [fixture().record] };
|
||||
const suites = historySuiteCatalog(history, runnerMatrix);
|
||||
expect(suites.find(suite => suite.id === "app-basics")).toMatchObject({ campaignCount: 1 });
|
||||
expect(suites.find(suite => suite.id === "core-compatibility")).toMatchObject({ campaignCount: 0 });
|
||||
expect(new Set(suites.map(suite => suite.id)).size).toBe(suites.length);
|
||||
});
|
||||
|
||||
it("uses the recorded full-selection verdict and denominator, independently of failures", () => {
|
||||
const suite = { ...fixture().record.suites[0]!, expected: 8, selected: 8, complete: true, passed: 5, failed: 3 };
|
||||
expect(fullSuiteSelection(suite)).toBe(true);
|
||||
expect(fullSuiteSelection({ ...suite, selected: 1, passed: 1, failed: 0 })).toBe(false);
|
||||
expect(fullSuiteSelection({ ...suite, complete: false })).toBe(false);
|
||||
expect(fullSuiteSelection({ ...suite, expected: 0, selected: 0 })).toBe(false);
|
||||
expect(historySuiteCoverage({ ...suite, executed: 8 })).toBe("full");
|
||||
expect(historySuiteCoverage({ ...suite, executed: 0 })).toBe("unfinished");
|
||||
expect(historySuiteCoverage({ ...suite, selected: 1 })).toBe("subset");
|
||||
expect(historySuiteCoverage({ ...suite, complete: false })).toBe("unknown");
|
||||
});
|
||||
|
||||
it("records target and agent execution separately without inventing historical targets", () => {
|
||||
const { record } = fixture();
|
||||
expect(record.executions[0]).toMatchObject({ targetUrl: "https://qa.test.paperclip.app/", runtimeMode: "none", environmentId: "daytona" });
|
||||
const row = historyBrowserCampaign(record);
|
||||
expect(row.executions[0]).toMatchObject({ target: "https://qa.test.paperclip.app", environment: "daytona", provider: "__none__", model: "__none__" });
|
||||
delete record.executions[0]!.targetUrl;
|
||||
delete record.executions[0]!.runtimeMode;
|
||||
expect(historyBrowserCampaign(record).executions[0]).toMatchObject({ target: "__unknown__", model: "__unknown__" });
|
||||
record.executions[0]!.targetUrl = "https://user:secret@example.test";
|
||||
expect(historyBrowserCampaign(record).executions[0]!.target).toBe("__unknown__");
|
||||
});
|
||||
|
||||
it("keeps public metadata inert inside the embedded JSON script", () => {
|
||||
const { record } = fixture(); record.source.ref = "</script><script>alert(1)</script>";
|
||||
const json = historyBrowserJson({ ...emptyRunnerHistory(), campaigns: [record] }, []);
|
||||
expect(json).not.toContain("<");
|
||||
expect(JSON.parse(json).campaigns[0].search).toContain(record.source.ref);
|
||||
});
|
||||
|
||||
it("imports a reviewed QA measurement as an ordinary campaign and preserves original files and results", async () => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "e2e-history-refresh-")); temporary.push(root);
|
||||
const original = JSON.stringify(emptyRunnerHistory());
|
||||
const { campaign } = fixture(); const measured = JSON.stringify(campaign);
|
||||
await writeFile(path.join(root, "history.json"), original); await writeFile(path.join(root, "campaign.json"), measured);
|
||||
const output = path.join(root, "output");
|
||||
const history = await refreshHistoryIndex({ history: path.join(root, "history.json"), output, publicCampaign: path.join(root, "campaign.json"), publicUrl: "https://pages.paperclip.ing/evals-qa-basics/", preset: "qa-smoke" });
|
||||
expect(history.campaigns[0]).toMatchObject({ campaignId: campaign.campaignId, generatedAt: campaign.generatedAt, source: campaign.source,
|
||||
suites: campaign.suites, selected: campaign.selected, passed: campaign.passed, preset: "qa-smoke" });
|
||||
expect(await readFile(path.join(root, "history.json"), "utf8")).toBe(original);
|
||||
expect(await readFile(path.join(root, "campaign.json"), "utf8")).toBe(measured);
|
||||
expect(await readFile(path.join(output, "index.html"), "utf8")).toContain('data-index-preset-link="qa-smoke"');
|
||||
await expect(refreshHistoryIndex({ history: path.join(output, "history.json"), output, publicCampaign: path.join(root, "campaign.json"), publicUrl: "https://pages.paperclip.ing/evals-qa-basics/" })).rejects.toThrow("already exists");
|
||||
});
|
||||
|
||||
it("refuses private Cloud bundles before creating a public history", async () => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "e2e-private-history-")); temporary.push(root);
|
||||
const { campaign } = fixture();
|
||||
campaign.results[0]!.productJourney = { privacy: "private" } as NonNullable<RunnerE2EResult["productJourney"]>;
|
||||
await writeFile(path.join(root, "history.json"), JSON.stringify(emptyRunnerHistory()));
|
||||
await writeFile(path.join(root, "private.json"), JSON.stringify(campaign));
|
||||
await expect(refreshHistoryIndex({ history: path.join(root, "history.json"), output: path.join(root, "output"),
|
||||
publicCampaign: path.join(root, "private.json"), publicUrl: "https://pages.paperclip.ing/private/" })).rejects.toThrow("Private Product E2E campaigns");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,217 @@
|
||||
import { retainedSuiteLabels, type ReportExecution } from "./report-catalog.js";
|
||||
import type { RunnerE2EHistoryCampaign, RunnerE2EHistoryIndex, RunnerE2ESuiteSummary } from "./types.js";
|
||||
|
||||
export function fullSuiteSelection(suite: RunnerE2ESuiteSummary) {
|
||||
return suite.complete && suite.expected > 0 && suite.selected === suite.expected;
|
||||
}
|
||||
|
||||
export function historySuiteCoverage(suite: RunnerE2ESuiteSummary) {
|
||||
if (fullSuiteSelection(suite)) return suite.executed === suite.selected ? "full" : "unfinished";
|
||||
return suite.selected < suite.expected ? "subset" : "unknown";
|
||||
}
|
||||
|
||||
export function historySuiteCatalog(history: RunnerE2EHistoryIndex, catalog: readonly ReportExecution[]) {
|
||||
const suites = new Map(catalog.map(row => [row.suite.id, { id: row.suite.id, label: row.suite.label }]));
|
||||
for (const campaign of history.campaigns) for (const suite of campaign.suites) {
|
||||
if (!suites.has(suite.suiteId)) suites.set(suite.suiteId, {
|
||||
id: suite.suiteId, label: retainedSuiteLabels[suite.suiteId] ?? suite.suiteId.replaceAll("-", " "),
|
||||
});
|
||||
}
|
||||
return [...suites.values()].map(suite => ({ ...suite,
|
||||
campaignCount: history.campaigns.filter(campaign => campaign.suites.some(row => row.suiteId === suite.id)).length,
|
||||
})).sort((a, b) => a.label.localeCompare(b.label));
|
||||
}
|
||||
|
||||
/** Only recorded metadata is searchable; old records keep an explicit unknown target. */
|
||||
export function historyBrowserCampaign(campaign: RunnerE2EHistoryCampaign) {
|
||||
return {
|
||||
id: campaign.campaignId, date: campaign.generatedAt, url: campaign.publicUrl,
|
||||
search: [campaign.campaignId, campaign.source.sha, campaign.source.ref].filter(Boolean).join(" ").toLowerCase(),
|
||||
complete: campaign.complete, selected: campaign.selected, executed: campaign.executed,
|
||||
passed: campaign.passed, failed: campaign.failed, incomplete: campaign.incomplete ?? 0,
|
||||
cleanup: campaign.cleanupPassed, preset: campaign.preset ?? "",
|
||||
coverage: campaign.complete && campaign.selected > 0
|
||||
? campaign.executed === campaign.selected ? "full" : "unfinished"
|
||||
: "subset",
|
||||
suites: campaign.suites.map(suite => ({ id: suite.suiteId, full: historySuiteCoverage(suite) === "full", coverage: historySuiteCoverage(suite),
|
||||
expected: suite.expected, selected: suite.selected, executed: suite.executed, passed: suite.passed,
|
||||
failed: suite.failed, incomplete: suite.incomplete ?? 0, cleanup: suite.cleanupPassed, hash: suite.suiteDefinitionHash })),
|
||||
executions: campaign.executions.map(execution => {
|
||||
let target = "__unknown__";
|
||||
try {
|
||||
const url = new URL(execution.targetUrl ?? "");
|
||||
if (["http:", "https:"].includes(url.protocol) && !url.username && !url.password) target = url.origin;
|
||||
} catch { /* Historical records did not retain the application URL. */ }
|
||||
return { suite: execution.suiteId, profile: execution.profileId, environment: execution.environmentId,
|
||||
provider: execution.provider ?? (execution.runtimeMode === "none" ? "__none__" : "__unknown__"),
|
||||
model: execution.model ?? (execution.runtimeMode === "none" ? "__none__" : "__unknown__"),
|
||||
runtime: execution.runtimeMode ?? "__unknown__", target, case: execution.caseId,
|
||||
status: execution.status, cleanup: execution.cleanup, executed: execution.attempt > 0 };
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
export function historyBrowserJson(history: RunnerE2EHistoryIndex, catalog: readonly ReportExecution[]) {
|
||||
return JSON.stringify({ suites: historySuiteCatalog(history, catalog),
|
||||
campaigns: [...history.campaigns].sort((a, b) => Date.parse(b.generatedAt) - Date.parse(a.generatedAt)).map(historyBrowserCampaign),
|
||||
}).replaceAll("<", "\\u003c").replaceAll("\u2028", "\\u2028").replaceAll("\u2029", "\\u2029");
|
||||
}
|
||||
|
||||
export const historyBrowserScript = String.raw`
|
||||
(() => {
|
||||
const data = JSON.parse(document.querySelector('[data-index-data]').textContent);
|
||||
const shell = document.querySelector('.report-shell');
|
||||
const rows = new Map([...document.querySelectorAll('[data-index-campaign]')].map(row => [row.dataset.indexId, row]));
|
||||
const suiteLabels = new Map(data.suites.map(suite => [suite.id, suite.label]));
|
||||
const names = ['suite','coverage','target','environment','provider','model','profile','runtime','preset','status','from','through','q'];
|
||||
const executionNames = ['target','environment','provider','model','profile','runtime'];
|
||||
const controls = Object.fromEntries(names.map(name => [name, document.querySelector('[data-index-filter="'+name+'"]')]));
|
||||
const unknown = value => value === '__none__' ? 'Browser only' : value === '__unknown__' ? 'Not recorded' : value;
|
||||
const label = (name, value) => name === 'suite' ? suiteLabels.get(value) || value :
|
||||
name === 'target' && value !== '__unknown__' ? (new URL(value).hostname === 'qa.test.paperclip.app' ? 'QA staging · ' : '') + value :
|
||||
name === 'preset' && value === 'qa-smoke' ? 'QA smoke' : unknown(value);
|
||||
for (const name of [...executionNames, 'preset']) {
|
||||
const values = [...new Set(data.campaigns.flatMap(campaign => name === 'preset' ? [campaign.preset].filter(Boolean) : campaign.executions.map(execution => execution[name])))].sort();
|
||||
for (const value of values) { const option = document.createElement('option'); option.value = value; option.textContent = label(name, value); controls[name].append(option); }
|
||||
}
|
||||
let state = {}, view = 'overview';
|
||||
const reportHref = campaign => {
|
||||
const url = new URL(campaign.url,location.href); url.searchParams.set('view','all');
|
||||
for (const name of ['suite','profile','environment']) if (state[name]) url.searchParams.set(name,state[name]);
|
||||
if (state.suite) url.hash = 'suite-' + state.suite;
|
||||
return url.href;
|
||||
};
|
||||
const status = counts => counts.failed > 0 || !counts.cleanup || counts.executed < counts.selected ? 'failed' : counts.incomplete > 0 ? 'incomplete' : 'passed';
|
||||
const selection = campaign => {
|
||||
const suite = state.suite ? campaign.suites.find(suite => suite.id === state.suite) : null;
|
||||
if (state.suite && !suite || state.preset && campaign.preset !== state.preset) return null;
|
||||
if (state.from && campaign.date.slice(0,10) < state.from || state.through && campaign.date.slice(0,10) > state.through) return null;
|
||||
const executions = campaign.executions.filter(execution => (!state.suite || execution.suite === state.suite) && executionNames.every(name => !state[name] || execution[name] === state[name]));
|
||||
const narrowed = executionNames.some(name => state[name]);
|
||||
if (narrowed && !executions.length) return null;
|
||||
const search = [campaign.search, ...executions.flatMap(execution => Object.values(execution)),
|
||||
...campaign.suites.filter(item => !state.suite || item.id === state.suite).map(item => suiteLabels.get(item.id))].join(' ').toLowerCase();
|
||||
if (state.q && !search.includes(state.q.trim().toLowerCase())) return null;
|
||||
const counts = narrowed ? { selected: executions.length, executed: executions.filter(execution => execution.executed).length,
|
||||
passed: executions.filter(execution => execution.status === 'passed' && execution.cleanup === 'passed').length,
|
||||
failed: executions.filter(execution => execution.status === 'failed').length,
|
||||
incomplete: executions.filter(execution => execution.status === 'incomplete').length,
|
||||
cleanup: executions.every(execution => execution.cleanup === 'passed') } : suite || campaign;
|
||||
const result = status(counts);
|
||||
if (state.status && result !== state.status || view === 'attention' && result === 'passed') return null;
|
||||
const coverage = suite ? suite.coverage : campaign.coverage;
|
||||
return { campaign, suite, counts, result, full: coverage === 'full', coverage, narrowed };
|
||||
};
|
||||
const fillPointer = (kind, selected) => {
|
||||
const root = document.querySelector('[data-index-latest="'+kind+'"]');
|
||||
const link = root.querySelector('a'), detail = root.querySelector('small');
|
||||
link.hidden = !selected;
|
||||
if (!selected) { detail.textContent = kind === 'full' ? 'No full run recorded for these filters.' : 'No matching run recorded.'; link.removeAttribute('href'); return; }
|
||||
link.href = reportHref(selected.campaign);
|
||||
link.textContent = selected.campaign.id;
|
||||
detail.textContent = new Date(selected.campaign.date).toLocaleString('en-US',{timeZone:'UTC'}) + ' UTC · ' +
|
||||
selected.counts.passed + '/' + selected.counts.selected + ' passed' + (selected.suite ? ' · ' + selected.suite.selected + '/' + selected.suite.expected + ' selected · definition ' + selected.suite.hash.slice(0,8) : '') +
|
||||
(kind === 'full' && state.coverage && state.coverage !== 'full' ? ' · across coverage selections' : '');
|
||||
};
|
||||
const apply = () => {
|
||||
shell.dataset.view = view;
|
||||
const activeCount = names.filter(name => state[name]).length;
|
||||
const active = activeCount > 0;
|
||||
for (const name of names) controls[name].dataset.filterActive = String(Boolean(state[name]));
|
||||
const activeLabel = document.querySelector('[data-index-active-count]');
|
||||
activeLabel.textContent = active ? activeCount + ' active' : 'None active';
|
||||
activeLabel.dataset.active = String(active);
|
||||
document.querySelector('[data-index-reset]').dataset.active = String(active);
|
||||
const moreCount = names.filter(name => !['suite','coverage','target','q'].includes(name) && state[name]).length;
|
||||
const moreLabel = document.querySelector('[data-index-more-count]');
|
||||
moreLabel.hidden = moreCount === 0;
|
||||
moreLabel.textContent = moreCount + ' active';
|
||||
const selectedLabel = state.suite ? suiteLabels.get(state.suite) || state.suite : '';
|
||||
document.querySelector('[data-index-title]').textContent = selectedLabel || (view === 'attention' ? 'Needs attention' : 'E2E tests');
|
||||
document.querySelector('[data-index-description]').textContent = state.suite ? 'Campaigns that included this suite. Full runs selected and executed the entire recorded suite definition; failed tests remain visible.' : 'Browse suites and retained campaigns across deployment targets, agent execution environments, and models.';
|
||||
document.querySelector('[data-index-overview]').hidden = view !== 'overview' || active;
|
||||
document.querySelector('[data-index-trends]').hidden = view !== 'history' || active;
|
||||
document.querySelector('.summary').hidden = active;
|
||||
document.querySelector('[data-index-suite-summary]').hidden = !state.suite;
|
||||
const candidates = data.campaigns.map(selection).filter(Boolean);
|
||||
fillPointer('any', candidates.find(row => !state.coverage || row.coverage === state.coverage));
|
||||
fillPointer('full', candidates.find(row => row.full));
|
||||
let visible = 0;
|
||||
for (const campaign of data.campaigns) rows.get(campaign.id).hidden = true;
|
||||
for (const selected of candidates) {
|
||||
if (state.coverage && state.coverage !== selected.coverage) continue;
|
||||
if (view === 'overview' && !active && visible >= 10) continue;
|
||||
const row = rows.get(selected.campaign.id); row.hidden = false; visible++;
|
||||
for (const link of row.querySelectorAll('.campaign-link,.open-cell > a')) link.href = reportHref(selected.campaign);
|
||||
const result = row.querySelector('[data-index-row-status]'); result.textContent = selected.result; result.className = 'status status-' + selected.result;
|
||||
row.querySelector('[data-index-row-counts]').textContent = selected.counts.passed + '/' + selected.counts.selected + (selected.narrowed ? ' matching executions passed' : state.suite ? ' suite executions passed' : ' passed') + ' · ' + selected.counts.executed + ' executed';
|
||||
const coverageLabel = {full:'Full run',subset:'Subset selection',unfinished:'Unfinished full selection',unknown:'Coverage not established'}[selected.coverage];
|
||||
row.querySelector('[data-index-row-coverage]').textContent = coverageLabel + (selected.suite ? ' · ' + selected.suite.selected + '/' + selected.suite.expected + ' selected' : '');
|
||||
}
|
||||
document.querySelector('[data-index-result]').textContent = visible + ' matching campaign' + (visible === 1 ? '' : 's') + (view === 'overview' && !active && candidates.length > 10 ? ' · Showing the 10 most recent; open Campaign history for all.' : '');
|
||||
document.querySelector('[data-index-empty]').hidden = visible !== 0;
|
||||
const presetHasSuite = data.campaigns.some(campaign => campaign.preset === state.preset && campaign.suites.some(suite => suite.id === state.suite));
|
||||
const incompatiblePreset = state.suite && state.preset && !presetHasSuite;
|
||||
document.querySelector('[data-index-empty-message]').textContent = incompatiblePreset
|
||||
? 'No ' + selectedLabel + ' campaigns are included in ' + label('preset', state.preset) + '.'
|
||||
: 'No campaigns match these filters.';
|
||||
const suiteRuns = document.querySelector('[data-index-empty-suite]');
|
||||
suiteRuns.hidden = !incompatiblePreset;
|
||||
if (incompatiblePreset) {
|
||||
suiteRuns.href = '?view=history&suite=' + encodeURIComponent(state.suite);
|
||||
suiteRuns.textContent = 'View ' + selectedLabel + ' runs';
|
||||
}
|
||||
controls.coverage.closest('label').querySelector('span').textContent = state.suite ? 'Suite coverage' : 'Campaign coverage';
|
||||
controls.coverage.setAttribute('aria-label',state.suite ? 'Suite coverage' : 'Campaign coverage');
|
||||
document.querySelectorAll('[data-index-suite-link]').forEach(link => { if (link.dataset.indexSuiteLink === state.suite) link.setAttribute('aria-current','page'); else link.removeAttribute('aria-current'); });
|
||||
document.querySelectorAll('[data-nav-view]').forEach(link => { if (link.dataset.navView === view && !state.suite) link.setAttribute('aria-current','page'); else link.removeAttribute('aria-current'); });
|
||||
document.querySelectorAll('[data-index-preset-link]').forEach(link => { if (link.dataset.indexPresetLink === state.preset) link.setAttribute('aria-current','page'); else link.removeAttribute('aria-current'); });
|
||||
document.querySelectorAll('[data-history-suite-trends]').forEach(section => { section.hidden = Boolean(state.suite) && section.dataset.historySuiteTrends !== state.suite; });
|
||||
const crumbs = document.querySelector('[data-nav-breadcrumbs]'); crumbs.replaceChildren();
|
||||
const home = document.createElement('a'); home.href = '?view=history'; home.textContent = 'E2E tests'; crumbs.append(home);
|
||||
const separator = document.createElement('span'); separator.textContent = '/'; separator.setAttribute('aria-hidden','true'); crumbs.append(separator);
|
||||
const current = document.createElement('span'); current.textContent = selectedLabel || (view === 'attention' ? 'Needs attention' : view === 'overview' ? 'Overview' : 'Campaign history'); current.setAttribute('aria-current','page'); crumbs.append(current);
|
||||
};
|
||||
const write = push => {
|
||||
const url = new URL(location.href); url.search = ''; url.searchParams.set('view',view);
|
||||
for (const name of names) if (state[name]) url.searchParams.set(name,state[name]);
|
||||
history[push ? 'pushState' : 'replaceState'](null,'',url); read();
|
||||
};
|
||||
const read = () => {
|
||||
const params = new URLSearchParams(location.search); view = ['history','attention'].includes(params.get('view')) ? params.get('view') : 'overview';
|
||||
state = Object.fromEntries(names.map(name => [name,params.get(name) || '']));
|
||||
for (const name of names) {
|
||||
const control = controls[name];
|
||||
if (control.tagName === 'SELECT' && state[name] && ![...control.options].some(option => option.value === state[name])) {
|
||||
const option = document.createElement('option'); option.value = state[name]; option.textContent = 'Unknown: ' + state[name]; control.append(option);
|
||||
}
|
||||
control.value = state[name];
|
||||
}
|
||||
if (names.filter(name => !['suite','coverage','target','q'].includes(name)).some(name => state[name])) document.querySelector('[data-index-more]').open = true;
|
||||
apply();
|
||||
};
|
||||
document.addEventListener('click', event => {
|
||||
if (event.defaultPrevented || event.button !== 0 || event.metaKey || event.ctrlKey || event.shiftKey || event.altKey) return;
|
||||
const link = event.target.closest('[data-index-suite-link],[data-index-preset-link],[data-nav-view]'); if (!link) return;
|
||||
event.preventDefault();
|
||||
if (link.hasAttribute('data-index-suite-link')) { state = { suite: link.dataset.indexSuiteLink }; view = 'history'; }
|
||||
else if (link.hasAttribute('data-index-preset-link')) { state = {}; state.preset = link.dataset.indexPresetLink; view = 'history'; }
|
||||
else { view = link.dataset.navView; if (view === 'overview') state = {}; }
|
||||
write(true); window.scrollTo({top:0,behavior:'instant'});
|
||||
});
|
||||
for (const name of names) controls[name].addEventListener(name === 'q' ? 'input' : 'change', () => { state[name] = controls[name].value; write(name !== 'q'); });
|
||||
document.querySelector('[data-index-reset]').addEventListener('click', () => {
|
||||
state = {}; view = 'history';
|
||||
const suiteSearch = document.querySelector('[data-index-suite-query]');
|
||||
suiteSearch.value = ''; suiteSearch.dataset.filterActive = 'false';
|
||||
document.querySelectorAll('.report-suite-catalog a').forEach(link => { link.hidden = false; });
|
||||
write(true);
|
||||
});
|
||||
const suiteSearch = document.querySelector('[data-index-suite-query]');
|
||||
suiteSearch.addEventListener('input', () => {
|
||||
suiteSearch.dataset.filterActive = String(Boolean(suiteSearch.value));
|
||||
for (const link of document.querySelectorAll('.report-suite-catalog a')) link.hidden = !link.textContent.toLowerCase().includes(suiteSearch.value.trim().toLowerCase());
|
||||
});
|
||||
window.addEventListener('popstate',read); read();
|
||||
})();
|
||||
`;
|
||||
@@ -1,7 +1,8 @@
|
||||
import { renderReportHistory, reportHistoryStyles, reportHistoryScript } from "./report-history.js";
|
||||
import { renderReportHistory, reportHistoryStyles } from "./report-history.js";
|
||||
import { renderReportShell, reportNavigationStyles, reportShellScript } from "./report-navigation.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { campaignSuiteSummaryStyles, renderCampaignSuiteSummary } from "./campaign-summary.js";
|
||||
import { historyBrowserJson, historyBrowserScript, historySuiteCatalog } from "./history-browser.js";
|
||||
import type {
|
||||
RunnerE2EHistoryCampaign,
|
||||
RunnerE2EHistoryIndex,
|
||||
@@ -73,14 +74,15 @@ function campaignRow(campaign: RunnerE2EHistoryCampaign) {
|
||||
)
|
||||
.join("");
|
||||
const billing = campaign.billing;
|
||||
return `<tr data-index-campaign data-index-status="${status}" data-index-search="${html([campaign.campaignId, campaign.source.ref, campaign.source.sha].join(" ").toLowerCase())}">
|
||||
return `<tr data-index-campaign data-index-id="${html(campaign.campaignId)}" data-index-status="${status}">
|
||||
<td data-label="Campaign">
|
||||
<a class="campaign-link" href="${html(campaign.publicUrl)}">${html(campaign.campaignId)}</a>
|
||||
<small>${html(date(campaign.generatedAt))} UTC</small>
|
||||
</td>
|
||||
<td data-label="Result">
|
||||
<span class="status status-${status}">${status}</span>
|
||||
<small>${campaign.passed}/${campaign.selected} passed · ${campaign.executed} executed</small>
|
||||
<span class="status status-${status}" data-index-row-status>${status}</span>
|
||||
<small data-index-row-counts>${campaign.passed}/${campaign.selected} passed · ${campaign.executed} executed</small>
|
||||
<small data-index-row-coverage>${campaign.complete ? "Full campaign" : "Partial campaign"}</small>
|
||||
</td>
|
||||
<td data-label="Source">${sourceCell(campaign)}</td>
|
||||
<td class="open-cell" data-label="Details"><a href="${html(campaign.publicUrl)}" aria-label="Open campaign ${html(campaign.campaignId)}">Open report →</a>
|
||||
@@ -99,9 +101,7 @@ function campaignRow(campaign: RunnerE2EHistoryCampaign) {
|
||||
export function renderRunnerHistoryIndex(
|
||||
history: RunnerE2EHistoryIndex,
|
||||
) {
|
||||
const campaigns = [...history.campaigns].sort((left, right) =>
|
||||
right.generatedAt.localeCompare(left.generatedAt),
|
||||
);
|
||||
const campaigns = [...history.campaigns].sort((left, right) => Date.parse(right.generatedAt) - Date.parse(left.generatedAt));
|
||||
const passed = campaigns.filter(
|
||||
(campaign) => campaignStatus(campaign) === "passed",
|
||||
).length;
|
||||
@@ -119,6 +119,9 @@ export function renderRunnerHistoryIndex(
|
||||
campaigns.length > 0
|
||||
? campaigns.map(campaignRow).join("")
|
||||
: `<tr><td class="empty" colspan="4">No campaigns have been published yet.</td></tr>`;
|
||||
const suites = historySuiteCatalog(history, runnerMatrix);
|
||||
const allLabels: Record<string, string> = { suite: "All suites", target: "All targets", environment: "All execution environments", provider: "All providers", model: "All models", profile: "All profiles", runtime: "All runtimes", preset: "All presets", status: "All results", coverage: "Any coverage" };
|
||||
const filter = (name: string, label: string, choices: readonly [string, string][] = []) => `<label><span>${label}</span><select data-index-filter="${name}" aria-label="${label}"><option value="">${allLabels[name]}</option>${choices.map(([value, text]) => `<option value="${html(value)}">${html(text)}</option>`).join("")}</select></label>`;
|
||||
|
||||
return `<!doctype html>
|
||||
<html lang="en">
|
||||
@@ -129,7 +132,7 @@ export function renderRunnerHistoryIndex(
|
||||
<meta name="theme-color" content="#ffffff" media="(prefers-color-scheme: light)">
|
||||
<meta name="theme-color" content="#141413" media="(prefers-color-scheme: dark)">
|
||||
<link rel="icon" href="assets/favicon-32x32.png" type="image/png">
|
||||
<title>Runner E2E Campaigns · Paperclip</title>
|
||||
<title>E2E tests · Paperclip</title>
|
||||
<style>
|
||||
@font-face { font-family: "Paperclip Inter"; src: url("assets/InterVariable.woff2") format("woff2"); font-style: normal; font-weight: 100 900; font-display: swap; }
|
||||
:root { color-scheme: light dark; --background:#fff; --foreground:#0a0a0a; --muted:#60666a; --border:#e5e5e5; --raised:#fafafa; --pass:#188a3c; --pass-bg:#dcfce7; --fail:#991b1b; --fail-bg:#fee2e2; --font:"Paperclip Inter",Inter,ui-sans-serif,system-ui,-apple-system,sans-serif; --mono:ui-monospace,SFMono-Regular,Menlo,monospace; }
|
||||
@@ -156,6 +159,30 @@ export function renderRunnerHistoryIndex(
|
||||
.pointers a { padding:8px 11px; border:1px solid var(--border); border-radius:7px; background:var(--raised); font-size:12px; text-decoration:none; }
|
||||
.pointers a:hover,.campaign-link:hover,.open-cell a:hover { text-decoration:underline; }
|
||||
.latest-campaign { margin-bottom:32px; }
|
||||
.index-filter-toolbar { display:flex; align-items:center; justify-content:space-between; gap:16px; margin:24px 0 14px; }
|
||||
.index-filter-toolbar > div { display:flex; align-items:baseline; gap:10px; }
|
||||
.index-filter-toolbar h2 { margin:0; font-size:15px; }
|
||||
.index-active-count { color:var(--muted); font-size:12px; }
|
||||
.index-active-count[data-active=true],.index-more-count { color:var(--report-filter-active); }
|
||||
.index-filters { display: grid; grid-template-columns: minmax(160px,1fr) minmax(140px,.7fr) minmax(180px,1fr) minmax(180px,1fr); gap: 12px; margin: 0 0 12px; }
|
||||
.index-filters label,.index-more label { display: grid; gap: 6px; min-width: 0; font-size: 11px; color: var(--muted); }
|
||||
.index-filters select,.index-filters input,.index-more select,.index-more input { width: 100%; min-width: 0; min-height: 38px; padding: 8px; background: var(--background); color: var(--foreground); border: 1px solid var(--border); border-radius: 5px; font: 12px var(--font); }
|
||||
.index-more { margin-bottom: 20px; }
|
||||
.index-more summary { color: var(--muted); cursor: pointer; font-size: 12px; }
|
||||
.index-more-grid { display: grid; grid-template-columns: repeat(3,minmax(0,1fr)); gap: 12px; margin-top: 12px; }
|
||||
.index-more-count { margin-left:8px; font-size:11px; }
|
||||
.index-reset { min-height:36px; padding:8px 12px; border:1px solid var(--border); border-radius:5px; background:var(--raised); color:var(--foreground); font:600 12px var(--font); cursor:pointer; white-space:nowrap; }
|
||||
.index-reset[data-active=true] { border-color:var(--report-filter-active); color:var(--report-filter-active); }
|
||||
.index-reset:hover { background:var(--report-filter-tint); }
|
||||
.index-reset:focus-visible { outline:2px solid var(--report-filter-active); outline-offset:3px; }
|
||||
.suite-run-pointers { display: grid; grid-template-columns: repeat(2,minmax(0,1fr)); gap: 16px; margin: 20px 0; }
|
||||
.suite-run-pointers > div { min-width: 0; border: 1px solid var(--border); border-radius: 7px; padding: 14px; }
|
||||
.suite-run-pointers strong,.suite-run-pointers a,.suite-run-pointers small { display: block; }
|
||||
.suite-run-pointers strong { font-size: 12px; margin-bottom: 8px; }
|
||||
.suite-run-pointers a { font: 11px var(--mono); overflow-wrap: anywhere; }
|
||||
.suite-run-pointers small { margin-top: 8px; color: var(--muted); font-size: 11px; }
|
||||
@media(max-width:1100px) { .index-filters,.index-more-grid { grid-template-columns:repeat(2,minmax(0,1fr)); } }
|
||||
@media(max-width:520px) { .index-filters,.index-more-grid,.suite-run-pointers { grid-template-columns:1fr; } .index-filters input,.index-filters select,.index-more input,.index-more select,.report-suite-catalog input { font-size:16px; } }
|
||||
.latest-campaign-heading { display:flex; flex-wrap:wrap; align-items:baseline; gap:12px; margin-bottom:20px; }
|
||||
.latest-campaign-heading h2 { margin:0; font-size:18px; font-weight:600; }
|
||||
.latest-metrics { display:flex; flex-wrap:wrap; gap:24px 48px; margin:0 0 16px; }
|
||||
@@ -168,7 +195,7 @@ export function renderRunnerHistoryIndex(
|
||||
tr:last-child td { border-bottom:0; }
|
||||
th { background:var(--raised); color:var(--muted); font-size:10px; letter-spacing:.06em; text-transform:uppercase; white-space:nowrap; }
|
||||
td strong,.campaign-link { font-weight:650; }
|
||||
td small { display:block; margin-top:4px; color:var(--muted); font-size:10px; }
|
||||
td small { display:block; margin-top:4px; color:var(--muted); font-size:10px; overflow-wrap:anywhere; }
|
||||
td a.secondary-link { display:block; margin-top:3px; }
|
||||
.campaign-link { font:600 11px/1.4 var(--mono); overflow-wrap:anywhere; }
|
||||
.status { display:inline-block; padding:3px 8px; border-radius:999px; font-size:10px; font-weight:700; letter-spacing:.05em; text-transform:uppercase; }
|
||||
@@ -209,16 +236,21 @@ export function renderRunnerHistoryIndex(
|
||||
.campaign-row-details .suite-list { white-space:normal; }
|
||||
${reportNavigationStyles}
|
||||
${reportHistoryStyles}
|
||||
@media(max-width:860px) { .report-surface table { min-width:0; } }
|
||||
@media(max-width:860px) {
|
||||
.report-surface table { min-width:0; }
|
||||
.report-surface td:not(.open-cell)::before { grid-column:1; grid-row:1 / span 4; }
|
||||
.report-surface td:not(.open-cell) > * { grid-column:2; min-width:0; }
|
||||
.report-surface td .status { justify-self:start; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
${renderReportShell({ history })}
|
||||
${renderReportShell({ history, historySuites: suites })}
|
||||
<main>
|
||||
<header>
|
||||
<div>
|
||||
<p class="eyebrow">Historical test reporting</p>
|
||||
<h1 data-index-title>Runner E2E campaigns</h1>
|
||||
<h1 data-index-title>E2E tests</h1>
|
||||
<p class="lede" data-index-description>${campaigns.length} retained campaigns. Open a report for test outcomes, per-test billing, declared screenshots, and normalized results.</p>
|
||||
</div>
|
||||
<div class="summary" aria-label="History summary">
|
||||
@@ -242,7 +274,25 @@ export function renderRunnerHistoryIndex(
|
||||
${latestGreen ? `<a href="${html(latestGreen.publicUrl)}">Latest complete green · ${html(latestGreen.campaignId)}</a>` : ""}
|
||||
</nav>
|
||||
</section>
|
||||
<div class="index-search"><h2 data-index-list-title>Recent campaigns</h2><input type="search" data-index-query aria-label="Search campaigns" placeholder="Find a campaign, revision, or branch…"></div>
|
||||
<section data-index-suite-summary hidden aria-label="Suite run pointers"><div class="suite-run-pointers">
|
||||
<div data-index-latest="any"><strong>Latest matching run</strong><a hidden></a><small></small></div>
|
||||
<div data-index-latest="full"><strong>Latest full run</strong><a hidden></a><small></small></div>
|
||||
</div></section>
|
||||
<div class="index-filter-toolbar"><div><h2>Filters</h2><span class="index-active-count" data-index-active-count aria-live="polite">None active</span></div><button class="index-reset" type="button" data-index-reset>Reset all filters</button></div>
|
||||
<div class="index-filters" aria-label="Filter campaigns">
|
||||
${filter("suite", "Suite", suites.map(suite => [suite.id, suite.label]))}
|
||||
${filter("coverage", "Campaign coverage", [["full", "Full runs"], ["subset", "Subset selections"], ["unfinished", "Unfinished full selections"], ["unknown", "Coverage not established"]])}
|
||||
${filter("target", "Target")}
|
||||
<label><span>Search campaigns</span><input type="search" data-index-filter="q" aria-label="Search campaigns" placeholder="Campaign, revision, branch, or case…"></label>
|
||||
</div>
|
||||
<details class="index-more" data-index-more><summary><span>More filters</span><span class="index-more-count" data-index-more-count hidden></span></summary><div class="index-more-grid">
|
||||
${filter("environment", "Agent execution")}${filter("provider", "Provider")}${filter("model", "Model")}
|
||||
${filter("profile", "Agent profile")}${filter("runtime", "Runtime")}${filter("preset", "Run preset")}
|
||||
${filter("status", "Result", [["passed", "Passed"], ["failed", "Failed"], ["incomplete", "Incomplete"]])}
|
||||
<label><span>From</span><input type="date" data-index-filter="from" aria-label="From"></label>
|
||||
<label><span>Through</span><input type="date" data-index-filter="through" aria-label="Through"></label>
|
||||
</div></details>
|
||||
<h2 data-index-list-title>Campaigns</h2>
|
||||
<p class="index-result" data-index-result aria-live="polite"></p>
|
||||
<div class="table-wrap">
|
||||
<table>
|
||||
@@ -250,39 +300,12 @@ export function renderRunnerHistoryIndex(
|
||||
<tbody>${rows}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<details data-index-trends hidden><summary>Trends and detailed history</summary>${renderReportHistory(history)}</details>
|
||||
<div class="empty" data-index-empty hidden><p data-index-empty-message>No campaigns match these filters.</p><a data-index-empty-suite hidden></a></div>
|
||||
<details data-index-trends hidden><summary>Campaign trends</summary>${renderReportHistory(history, { chartsOnly: true })}</details>
|
||||
<footer><span>Updated ${html(date(history.updatedAt))} UTC</span><span>Immutable campaign reports · Declared screenshots and normalized results</span></footer>
|
||||
</main></div></div>
|
||||
<script>${reportShellScript}${reportHistoryScript}
|
||||
(() => {
|
||||
const shell = document.querySelector('.report-shell');
|
||||
const rows = [...document.querySelectorAll('[data-index-campaign]')];
|
||||
const query = document.querySelector('[data-index-query]');
|
||||
const crumbs = document.querySelector('[data-nav-breadcrumbs]');
|
||||
const originalDescription = document.querySelector('[data-index-description]').textContent;
|
||||
const apply = () => {
|
||||
const params = new URLSearchParams(location.search);
|
||||
const view = ['history','attention'].includes(params.get('view')) ? params.get('view') : 'overview';
|
||||
const title = {overview:'Runner E2E campaigns',history:'Campaign history',attention:'Needs attention'}[view];
|
||||
shell.dataset.view = view;
|
||||
document.querySelector('[data-index-title]').textContent = title;
|
||||
document.querySelector('[data-index-overview]').hidden = view !== 'overview';
|
||||
document.querySelector('[data-index-trends]').hidden = view !== 'history';
|
||||
document.querySelector('[data-index-description]').textContent = view === 'attention' ? 'Campaigns with failed or incomplete executions, missing results, or cleanup failures.' : originalDescription;
|
||||
document.querySelector('[data-index-list-title]').textContent = view === 'overview' ? 'Recent campaigns' : 'Campaigns';
|
||||
crumbs.replaceChildren();
|
||||
const root = document.createElement('a'); root.href = '?view=history'; root.textContent = 'Campaigns'; root.dataset.navView = 'history'; crumbs.append(root);
|
||||
const separator = document.createElement('span');separator.setAttribute('aria-hidden','true');separator.textContent='/';crumbs.append(separator);
|
||||
const current = document.createElement('span');current.setAttribute('aria-current','page');current.textContent = view === 'overview' ? 'Overview' : title;crumbs.append(current);
|
||||
document.querySelectorAll('[data-nav-view]').forEach(link => { if(link.dataset.navView===view)link.setAttribute('aria-current','page');else link.removeAttribute('aria-current'); });
|
||||
const search=query.value.trim().toLowerCase(); let visible=0,matching=0;
|
||||
rows.forEach(row => { const match=(!search||row.dataset.indexSearch.includes(search))&&(view!=='attention'||row.dataset.indexStatus!=='passed');if(match)matching++;row.hidden=!match||(view==='overview'&&!search&&visible>=10);if(!row.hidden)visible++; });
|
||||
document.querySelector('[data-index-result]').textContent=visible+' of '+matching+' campaigns'+(view==='overview'&&!search&&matching>10?' · Open Campaign history for all retained reports':'');
|
||||
};
|
||||
document.addEventListener('click',event=>{ const link=event.target.closest('[data-nav-view]');if(!link||event.defaultPrevented||event.button!==0||event.metaKey||event.ctrlKey||event.shiftKey||event.altKey)return;event.preventDefault();const url=new URL(location.href);url.searchParams.set('view',link.dataset.navView);url.searchParams.delete('q');history.pushState(null,'',url);query.value='';apply();window.scrollTo({top:0,behavior:'instant'}); });
|
||||
query.addEventListener('input',()=>{const url=new URL(location.href);if(query.value)url.searchParams.set('q',query.value);else url.searchParams.delete('q');history.replaceState(null,'',url);apply();});
|
||||
const read=()=>{query.value=new URLSearchParams(location.search).get('q')||'';apply();};window.addEventListener('popstate',read);read();
|
||||
})();
|
||||
<script type="application/json" data-index-data>${historyBrowserJson(history, runnerMatrix)}</script>
|
||||
<script>${reportShellScript}${historyBrowserScript}
|
||||
</script>
|
||||
</body>
|
||||
</html>`;
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
import path from "node:path";
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { parseArgs } from "node:util";
|
||||
import { renderRunnerHistoryIndex } from "./history-index.js";
|
||||
import { campaignHistoryRecord, mergeRunnerHistory } from "./history.js";
|
||||
import { assertPublicCampaign, buildHistoryPointers } from "./history-publish.js";
|
||||
import { validateRetainedRunnerResult } from "./result-validation.js";
|
||||
import type { RunnerE2ECampaign, RunnerE2EHistoryIndex } from "./types.js";
|
||||
|
||||
/** Index-only refresh: no provider calls, campaign rewrites, or evidence uploads. */
|
||||
export async function refreshHistoryIndex(input: {
|
||||
history: string; output: string; publicCampaign?: string; publicUrl?: string; preset?: string;
|
||||
}) {
|
||||
let history = JSON.parse(await readFile(input.history, "utf8")) as RunnerE2EHistoryIndex;
|
||||
if (history.schema !== "paperclip.runner-e2e.history/v1" || !Array.isArray(history.campaigns)) throw new Error("Invalid E2E history");
|
||||
if (input.publicCampaign) {
|
||||
const url = new URL(input.publicUrl ?? "");
|
||||
if (url.protocol !== "https:" || url.username || url.password || url.search || url.hash) throw new Error("A public HTTPS campaign URL is required");
|
||||
if (input.preset && !/^[a-z0-9][a-z0-9-]{0,63}$/.test(input.preset)) throw new Error("Invalid run preset");
|
||||
const campaign = JSON.parse(await readFile(input.publicCampaign, "utf8")) as RunnerE2ECampaign;
|
||||
if (campaign.schema !== "paperclip.runner-e2e.campaign/v2" || !Array.isArray(campaign.results)) throw new Error("A reviewed public campaign is required");
|
||||
assertPublicCampaign(campaign);
|
||||
for (const result of campaign.results) validateRetainedRunnerResult(result);
|
||||
const record = { ...campaignHistoryRecord(campaign, url.origin), publicUrl: url.href,
|
||||
...(input.preset ? { preset: input.preset } : {}) };
|
||||
if (history.campaigns.some(existing => existing.campaignId === record.campaignId)) {
|
||||
throw new Error("Campaign already exists in history; existing measurements are never replaced by an index refresh");
|
||||
}
|
||||
history = mergeRunnerHistory(history, record);
|
||||
} else if (input.publicUrl || input.preset) throw new Error("Campaign metadata requires a reviewed public campaign");
|
||||
const html = renderRunnerHistoryIndex(history);
|
||||
const pointers = buildHistoryPointers(history);
|
||||
await mkdir(input.output, { recursive: true });
|
||||
await Promise.all([
|
||||
writeFile(path.join(input.output, "index.html"), html),
|
||||
writeFile(path.join(input.output, "history.json"), JSON.stringify(history, null, 2) + "\n"),
|
||||
writeFile(path.join(input.output, "latest.json"), JSON.stringify(pointers.latest, null, 2) + "\n"),
|
||||
writeFile(path.join(input.output, "latest-green.json"), JSON.stringify(pointers.latestGreen, null, 2) + "\n"),
|
||||
]);
|
||||
return history;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === path.resolve(import.meta.filename)) {
|
||||
const { values } = parseArgs({ options: { history: { type: "string" }, output: { type: "string" },
|
||||
"public-campaign": { type: "string" }, "public-url": { type: "string" }, preset: { type: "string" } } });
|
||||
if (!values.history || !values.output) throw new Error("--history and --output are required");
|
||||
const history = await refreshHistoryIndex({ history: values.history, output: values.output,
|
||||
publicCampaign: values["public-campaign"], publicUrl: values["public-url"], preset: values.preset });
|
||||
console.log(`Rendered E2E history with ${history.campaigns.length} retained campaigns; no provider calls.`);
|
||||
}
|
||||
@@ -179,7 +179,7 @@ describe("runner E2E campaign history", () => {
|
||||
expect(history.latestGreenCampaignId).toBeNull();
|
||||
expect(history.latestGreenBySuite["first-task"]).toBeUndefined();
|
||||
expect(renderRunnerHistoryIndex(history)).toContain("1 incomplete");
|
||||
expect(renderRunnerHistoryIndex(history)).toContain('data-history-status="incomplete"');
|
||||
expect(renderRunnerHistoryIndex(history)).toContain('data-index-status="incomplete"');
|
||||
});
|
||||
|
||||
it("shows unknown campaign costs without plotting zero dollars", () => {
|
||||
@@ -319,8 +319,8 @@ describe("runner E2E campaign history", () => {
|
||||
expect(dashboard).not.toContain('class="history-section"');
|
||||
const index = renderRunnerHistoryIndex(history);
|
||||
expect(index).toContain("Campaign trends");
|
||||
expect(index).toContain("data-history-from");
|
||||
expect(index).toContain("data-history-through");
|
||||
expect(index).toContain('data-index-filter="from"');
|
||||
expect(index).toContain('data-index-filter="through"');
|
||||
expect(index).toContain(
|
||||
'data-history-suite-trends="core-compatibility"',
|
||||
);
|
||||
@@ -333,7 +333,7 @@ describe("runner E2E campaign history", () => {
|
||||
expect(index).toContain("Suite pass rate");
|
||||
expect(index).toContain("lines break at definition changes");
|
||||
expect(index).toContain("cleanup passed");
|
||||
expect(index).toContain("Runner E2E campaigns");
|
||||
expect(index).toContain("E2E tests");
|
||||
expect(index).toContain("complete-green");
|
||||
expect(index).toContain("complete-red");
|
||||
expect(index).toContain(`${runnerMatrix.length}/${runnerMatrix.length} passed`);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { isIncompleteFirstTaskResult } from "./case-outcome.js";
|
||||
import { reportTargetUrl } from "./report-target.js";
|
||||
import { buildCampaignAccounting } from "./attempt-accounting.js";
|
||||
import { validateEvalOperationsReport } from "./operations-report.js";
|
||||
import { validateCampaignFeedbackReview } from "./feedback-dispositions.js";
|
||||
@@ -269,6 +270,8 @@ export function campaignHistoryRecord(
|
||||
caseId: result.caseId,
|
||||
provider: result.provider,
|
||||
model: result.model,
|
||||
...(reportTargetUrl(result) ? { targetUrl: reportTargetUrl(result)! } : {}),
|
||||
runtimeMode: result.runtimeMode,
|
||||
status: isIncompleteFirstTaskResult(result) ? "incomplete" : result.status,
|
||||
durationMs: result.durationMs,
|
||||
attempt: result.attempt,
|
||||
|
||||
@@ -54,6 +54,7 @@ export const reportBrowserScript = String.raw`
|
||||
} else appendCrumb(view === 'attention' ? 'Needs attention' : view === 'case' ? 'Case unavailable' : 'All results');
|
||||
};
|
||||
const apply = () => {
|
||||
controls.forEach(control => { control.dataset.filterActive = String(Boolean(control.value)); });
|
||||
const search = query.value.trim().toLowerCase();
|
||||
const matchesFilters = card =>
|
||||
(!search || card.dataset.reportSearch.includes(search)) &&
|
||||
|
||||
@@ -2,6 +2,9 @@ import { validateRetainedRunnerResult } from "./result-validation.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import type { MatrixExecution, RunnerE2EResult } from "./types.js";
|
||||
|
||||
/** Presentation aliases only; retained execution identities never change. */
|
||||
export const retainedSuiteLabels: Readonly<Record<string, string>> = { "qa-basics": "Basic app journeys" };
|
||||
|
||||
/** Display-only metadata: never load or reconstruct executable branch fixtures. */
|
||||
export interface ReportExecution {
|
||||
id: string;
|
||||
@@ -92,7 +95,7 @@ export function discoverReportCatalog(input: {
|
||||
.sort();
|
||||
const suite = knownSuites.get(identity.suiteId) ?? {
|
||||
id: identity.suiteId,
|
||||
label: identity.suiteId,
|
||||
label: retainedSuiteLabels[identity.suiteId] ?? identity.suiteId,
|
||||
description:
|
||||
"Suite discovered from retained campaign identities; full suite size is not known to this publisher.",
|
||||
expectedMatrixSize: suiteIds.length,
|
||||
|
||||
@@ -67,7 +67,7 @@ function suiteSummaryFor(
|
||||
return campaign.suites.find((suite) => suite.suiteId === suiteId);
|
||||
}
|
||||
|
||||
export function renderReportHistory(history: RunnerE2EHistoryIndex | undefined) {
|
||||
export function renderReportHistory(history: RunnerE2EHistoryIndex | undefined, options: { chartsOnly?: boolean } = {}) {
|
||||
if (!history || history.campaigns.length === 0) {
|
||||
return `<section class="history-section" id="history"><div class="section-heading"><div><p class="eyebrow">History</p><h2>Campaign trends</h2></div><p>No historical campaigns have been published yet.</p></div></section>`;
|
||||
}
|
||||
@@ -211,7 +211,7 @@ export function renderReportHistory(history: RunnerE2EHistoryIndex | undefined)
|
||||
</nav>
|
||||
<div class="trend-grid">${charts}</div>
|
||||
${suiteCharts}
|
||||
<div class="history-filters">
|
||||
${options.chartsOnly ? "" : `<div class="history-filters">
|
||||
<label>Search <input type="search" data-history-query placeholder="SHA, model, profile, case"></label>
|
||||
<label>Suite <select data-history-suite><option value="">All suites</option>${suiteIds.map((suiteId) => `<option value="${html(suiteId)}">${html(suiteId)}</option>`).join("")}</select></label>
|
||||
<label>Status <select data-history-status><option value="">All statuses</option><option value="passed">Passed</option><option value="failed">Failed</option><option value="incomplete">Incomplete</option></select></label>
|
||||
@@ -220,7 +220,7 @@ export function renderReportHistory(history: RunnerE2EHistoryIndex | undefined)
|
||||
<label class="history-checkbox"><input type="checkbox" data-history-partial> Include partial campaigns</label>
|
||||
</div>
|
||||
<div class="history-table-wrap"><table class="history-table"><thead><tr><th>Campaign</th><th>Paperclip SHA</th><th>Result</th><th>Tokens</th><th>Cost</th><th>Execution</th></tr></thead><tbody>${rows}</tbody></table></div>
|
||||
<p class="history-empty" data-history-empty hidden>No campaigns match these filters.</p>
|
||||
<p class="history-empty" data-history-empty hidden>No campaigns match these filters.</p>`}
|
||||
</section>`;
|
||||
}
|
||||
|
||||
|
||||
@@ -14,7 +14,8 @@ describe("evaluation report navigation", () => {
|
||||
expect(sidebar).toContain("Overview");
|
||||
expect(sidebar).toContain("Needs attention");
|
||||
expect(sidebar).toContain("Campaign history");
|
||||
expect(sidebar).not.toContain("Suites");
|
||||
expect(sidebar).toContain('aria-label="E2E suites"');
|
||||
expect(sidebar).toContain('data-index-suite-link="core-compatibility"');
|
||||
expect(page).not.toContain('aria-label="Campaign suites"');
|
||||
expect(page).not.toContain('data-nav-campaign aria-label="Campaign"');
|
||||
});
|
||||
@@ -42,9 +43,9 @@ describe("evaluation report navigation", () => {
|
||||
const history = emptyRunnerHistory();
|
||||
history.campaigns = [{campaignId:"campaign-a",publicUrl:"/custom/reports/campaigns/campaign-a/"} as typeof history.campaigns[number]];
|
||||
expect(reportHistoryHref(history,"campaign-a")).toBe("/custom/reports/");
|
||||
expect(reportHistoryHref(undefined,"campaign-a")).toBe("https://pages.paperclip.ing/evals/");
|
||||
expect(reportHistoryHref(undefined,"campaign-a")).toBe("https://d1p6rlowie26tp.cloudfront.net/runner-e2e/");
|
||||
history.campaigns[0]!.publicUrl = "https://pages.paperclip.ing/standalone-report/";
|
||||
expect(reportHistoryHref(history,"campaign-a")).toBe("https://pages.paperclip.ing/evals/");
|
||||
expect(reportHistoryHref(history,"campaign-a")).toBe("https://d1p6rlowie26tp.cloudfront.net/runner-e2e/");
|
||||
expect(page).toContain('class="brand-lockup report-brand" href="https://pages.paperclip.ing/evals/"');
|
||||
});
|
||||
});
|
||||
@@ -12,13 +12,14 @@ export function reportHistoryHref(history: RunnerE2EHistoryIndex | undefined, ca
|
||||
const url = history?.campaigns.find(campaign => campaign.campaignId === campaignId)?.publicUrl;
|
||||
return url && /campaigns\/[^/]+\/(?:index\.html)?$/.test(url)
|
||||
? url.replace(/campaigns\/[^/]+\/(?:index\.html)?$/, "")
|
||||
: "https://pages.paperclip.ing/evals/";
|
||||
: "https://d1p6rlowie26tp.cloudfront.net/runner-e2e/";
|
||||
}
|
||||
|
||||
export function renderReportShell(input: {
|
||||
campaignId?: string;
|
||||
catalog?: readonly ReportExecution[];
|
||||
history?: RunnerE2EHistoryIndex;
|
||||
historySuites?: readonly { id: string; label: string; campaignCount: number }[];
|
||||
}) {
|
||||
const h = reportHtml;
|
||||
const campaign = Boolean(input.catalog);
|
||||
@@ -28,7 +29,8 @@ export function renderReportShell(input: {
|
||||
const tree = campaign ? `<nav class="report-suite-tree" aria-label="Campaign suites"><p class="report-side-label">Suites</p>${suites.map(suite => {
|
||||
const cases = [...new Map(input.catalog!.filter(row => row.suite.id === suite.id).map(row => [row.task.id, row])).values()];
|
||||
return `<details data-nav-suite="${h(suite.id)}"><summary><a href="#suite-${h(suite.id)}" data-nav-suite-link="${h(suite.id)}">${h(suite.label)}<small>${cases.length}</small></a></summary><ul>${cases.map(row => `<li><a href="?view=case&case=${encodeURIComponent(`${suite.id}.${row.task.id}`)}#execution-${h(row.id)}" data-nav-case="${h(`${suite.id}.${row.task.id}`)}" data-report-case-link="${h(`${suite.id}.${row.task.id}`)}"><span class="case-status-dot" aria-hidden="true"></span><span>${h(row.task.label)}</span></a></li>`).join("")}</ul></details>`;
|
||||
}).join("")}</nav>` : "";
|
||||
}).join("")}</nav>` : input.historySuites ? `<nav class="report-suite-catalog" aria-label="E2E suites"><p class="report-side-label">Suites</p><input type="search" data-index-suite-query aria-label="Find suites" placeholder="Find suites…">${input.historySuites.map(suite => `<a href="?view=history&suite=${encodeURIComponent(suite.id)}" data-index-suite-link="${h(suite.id)}"><span>${h(suite.label)}</span><small title="${suite.campaignCount} campaigns">${suite.campaignCount}</small></a>`).join("")}</nav>` : "";
|
||||
const presets = [...new Set(input.history?.campaigns.map(row => row.preset).filter((preset): preset is string => Boolean(preset)) ?? [])];
|
||||
return `<div class="report-shell" data-report-scope="${campaign ? "campaign" : "global"}" data-campaign-label="${h(label)}">
|
||||
<button class="report-drawer-backdrop" type="button" data-nav-close aria-label="Close navigation" hidden></button>
|
||||
<aside class="report-sidebar" id="report-navigation" aria-label="Report navigation">
|
||||
@@ -37,7 +39,7 @@ export function renderReportShell(input: {
|
||||
<a href="${campaign ? "?view=all" : "?view=overview"}" data-nav-view="${campaign ? "all" : "overview"}" aria-current="page"><span aria-hidden="true">◫</span>${campaign ? "All results" : "Overview"}</a>
|
||||
<a href="?view=attention" data-nav-view="attention"><span aria-hidden="true">○</span>Needs attention</a>
|
||||
<a href="${h(root)}?view=history" ${campaign ? "" : 'data-nav-view="history"'}><span aria-hidden="true">↶</span>Campaign history</a>
|
||||
</nav>${tree}</div><div class="report-sidebar-footer">Product E2E${campaign ? `<span>${input.catalog!.length} executions</span>` : ""}</div>
|
||||
</nav>${!campaign && presets.length ? `<nav class="report-global-nav" aria-label="Run presets"><p class="report-side-label">Run presets</p>${presets.map(preset => `<a href="?view=history&preset=${encodeURIComponent(preset)}" data-index-preset-link="${h(preset)}">${h(preset === "qa-smoke" ? "QA smoke" : preset)}</a>`).join("")}</nav>` : ""}${tree}</div><div class="report-sidebar-footer">E2E tests${campaign ? `<span>${input.catalog!.length} executions</span>` : ""}</div>
|
||||
</aside><div class="report-surface">
|
||||
<header class="report-topbar"><button type="button" class="report-menu-toggle" data-nav-toggle aria-label="Open navigation" aria-controls="report-navigation" aria-expanded="false">☰</button>
|
||||
<nav class="report-breadcrumbs" aria-label="Breadcrumb" data-nav-breadcrumbs><a href="${h(root)}?view=history">Campaigns</a>${campaign ? `<span aria-hidden="true">/</span><span>${h(label)}</span><span aria-hidden="true">/</span><span aria-current="page">All results</span>` : '<span aria-hidden="true">/</span><span aria-current="page">Overview</span>'}</nav>
|
||||
@@ -61,6 +63,10 @@ export const reportShellScript = `
|
||||
|
||||
export const reportNavigationStyles = `
|
||||
:root { --report-sidebar-width: 250px; --navigation-sticky-offset: 64px; }
|
||||
:root { --report-filter-active:#2563eb; --report-filter-tint:#eff6ff; }
|
||||
@media(prefers-color-scheme:dark) { :root { --report-filter-active:#60a5fa; --report-filter-tint:#172235; } }
|
||||
.report-shell [data-filter-active=true] { border-color:var(--report-filter-active); background-color:var(--report-filter-tint); box-shadow:0 0 0 1px var(--report-filter-active); }
|
||||
.report-shell label:has([data-filter-active=true]) > span { color:var(--report-filter-active); }
|
||||
html { scroll-behavior: auto; }
|
||||
[hidden] { display: none !important; }
|
||||
.report-shell { min-height: 100dvh; }
|
||||
@@ -76,6 +82,12 @@ export const reportNavigationStyles = `
|
||||
.report-global-nav a:hover,.report-suite-tree a:hover { background: var(--raised); color: var(--foreground); }
|
||||
.report-global-nav a[aria-current=page] { color: var(--foreground); background: var(--raised); }
|
||||
.report-side-label { padding: 18px 10px 7px; margin: 0; font: 10px var(--font-mono); color: var(--muted-foreground); text-transform: uppercase; letter-spacing: .08em; }
|
||||
.report-suite-catalog { display: grid; gap: 2px; }
|
||||
.report-suite-catalog input { min-width: 0; width: calc(100% - 12px); margin: 0 6px 7px; padding: 8px; border: 1px solid var(--border); border-radius: 5px; color: var(--foreground); background: var(--background); font: 12px var(--font-sans); }
|
||||
.report-suite-catalog a { display: flex; align-items: baseline; justify-content: space-between; gap: 10px; padding: 8px 10px; color: var(--muted-foreground); text-decoration: none; border-radius: 5px; font-size: 12px; }
|
||||
.report-suite-catalog a span { min-width: 0; overflow-wrap: anywhere; }
|
||||
.report-suite-catalog a small { flex: none; font: 10px var(--font-mono); }
|
||||
.report-suite-catalog a:hover,.report-suite-catalog a[aria-current=page] { color: var(--foreground); background: var(--raised); }
|
||||
.report-suite-tree details > summary { display: flex; align-items: center; gap: 4px; list-style: none; cursor: pointer; padding-left: 4px; }
|
||||
.report-suite-tree summary::-webkit-details-marker { display: none; }
|
||||
.report-suite-tree summary::before { content: '›'; width: 12px; flex: none; color: var(--muted-foreground); }
|
||||
@@ -94,6 +106,15 @@ export const reportNavigationStyles = `
|
||||
.report-breadcrumbs [aria-current=page] { color: var(--foreground); }
|
||||
.report-breadcrumbs a,.report-breadcrumbs span { overflow-wrap: anywhere; }
|
||||
.report-campaign-select { font: 11px var(--font-sans); padding: 8px; color: var(--foreground); background: var(--card); border: 1px solid var(--border); border-radius: 5px; width: 220px; max-width: 30%; flex: none; }
|
||||
.report-shell .report-surface select {
|
||||
appearance: none;
|
||||
-webkit-appearance: none;
|
||||
padding-inline-end: 34px;
|
||||
background-image: linear-gradient(45deg, transparent 50%, var(--muted-foreground) 50%), linear-gradient(135deg, var(--muted-foreground) 50%, transparent 50%);
|
||||
background-position: calc(100% - 16px) center, calc(100% - 12px) center;
|
||||
background-size: 4px 4px;
|
||||
background-repeat: no-repeat;
|
||||
}
|
||||
.report-menu-toggle { display: none; border: 0; padding: 0; background: transparent; color: var(--foreground); font-size: 20px; cursor: pointer; }
|
||||
.report-drawer-backdrop { display: none; }
|
||||
.report-header h1 { font-size: 32px; }
|
||||
|
||||
@@ -398,7 +398,7 @@ describe("runner E2E report aggregation", () => {
|
||||
path.join(output, "dashboard.html"),
|
||||
"utf8",
|
||||
);
|
||||
expect(dashboard).toContain("Runner Full-Stack E2E");
|
||||
expect(dashboard).toContain("E2E tests");
|
||||
expect(dashboard).toMatch(/\$0\.0250*<\/strong><span>LLM reported subtotal/);
|
||||
expect(dashboard).toContain("<strong>2s</strong><span>Summed attempt time</span>");
|
||||
expect(dashboard).toContain("Carried-forward evidence can span older runs");
|
||||
|
||||
@@ -363,7 +363,7 @@ async function main() {
|
||||
`${JSON.stringify(normalized, null, 2)}\n`,
|
||||
);
|
||||
const dashboard = renderRunnerE2EDashboard({
|
||||
title: "Runner Full-Stack E2E",
|
||||
title: "E2E tests",
|
||||
generatedAt: normalized.generatedAt,
|
||||
expected,
|
||||
catalog: runnerMatrix,
|
||||
@@ -393,7 +393,7 @@ async function main() {
|
||||
)?.url;
|
||||
|
||||
const summaryLines = [
|
||||
"# Runner Full-Stack E2E",
|
||||
"# E2E tests",
|
||||
"",
|
||||
`Passed: ${normalized.passed}/${selected.length}`,
|
||||
"",
|
||||
|
||||
@@ -3342,6 +3342,7 @@ for (const execution of executions) {
|
||||
...(completionQuality.length ? { completionQuality } : {}),
|
||||
...(publicMcpUsage ? { publicMcp: publicMcpUsage } : {}),
|
||||
runtimeMode: execution.profile.expectedRuntimeMode,
|
||||
targetUrl: testInfo.project.use.baseURL,
|
||||
issueId: issue?.id,
|
||||
issueIdentifier: issue?.identifier ?? null,
|
||||
runIds: selectedRuns.map((run) => run.id),
|
||||
|
||||
@@ -477,6 +477,9 @@ export interface RunnerE2EHistoryExecution {
|
||||
caseId: string;
|
||||
provider: string | null;
|
||||
model: string | null;
|
||||
/** Absent in older records; never infer the application from the worker. */
|
||||
targetUrl?: string;
|
||||
runtimeMode?: RunnerE2EResult["runtimeMode"];
|
||||
status: "passed" | "failed" | "incomplete";
|
||||
durationMs: number;
|
||||
attempt: number;
|
||||
@@ -500,6 +503,8 @@ export interface RunnerE2EHistoryCampaign {
|
||||
billing: RunnerE2EAggregateBillingSummary;
|
||||
suites: RunnerE2ESuiteSummary[];
|
||||
executions: RunnerE2EHistoryExecution[];
|
||||
/** A named case selection, independent of suite and deployment target. */
|
||||
preset?: string;
|
||||
}
|
||||
|
||||
export interface RunnerE2EHistoryIndex {
|
||||
|
||||
Reference in new issue
Block a user