Files
PaperClipAI/tests/runner-e2e/campaign-summary.ts
T
DottaandPaperclip 2b16262280 Improve runner E2E report navigation and local previews (#16996)
## Thinking Path

> - Paperclip is the open source app that people use to manage AI agents
for work.
> - Product E2E campaigns check user journeys across agent profiles and
environments.
> - Readers use the campaign history and reports to inspect results and
retained evidence.
> - The reports showed image summaries, many unselected suites, and a
long sticky index.
> - Individual case cards also showed routine diagnostics before the
visual evidence.
> - This pull request adds text summaries, feature filters, folding case
indexes, and simpler cards.
> - A local preview uses retained public data so designers can change
reports without another paid campaign.

## Linked Issues or Issue Description

**Current behavior**

The report is hard to scan as the journey catalog grows. Suite links can
point into hidden results. Case indexes do not show whether a case
passed across its configurations. Design changes also need a realistic
local preview.

**Proposed behavior**

Show the selected campaign as text. Link journeys to current feature-map
capabilities. Add folding case indexes with status indicators, reliable
section links, and a back-to-top control. Keep failed outcomes visible
and put routine diagnostics under Test details. Render retained public
campaign data with local templates.

**Reason and benefit**

Readers can find a feature or failed case faster. Designers can iterate
against real retained data without starting an application instance or
using provider credentials.

**Breaking changes**

No result schema or grading change. The report stops embedding the
generated summary PNG. The publisher still produces that artifact for
existing consumers.

Related work: #12889 added the existing matrix browser. #12846 concerns
the public image publication boundary. This PR changes report
presentation and local preview behavior.

## What Changed

- Replace image summaries with shared HTML suite summaries on the
history and campaign pages.
- Show selected executions and preserve missing results for selected
cells.
- Add explicit journey-to-capability associations, feature filters, and
a coverage view with unmapped journeys.
- Add folding case indexes, section case lists, and red, green, or amber
status indicators across matching configurations.
- Fix anchor navigation through filters and add a floating back-to-top
control.
- Simplify case cards. Keep failure reasons visible. Fold routine
checks, billing, and context into Test details.
- Add an original-image link to the gallery.
- Add a local preview utility with public data caching and offline mode.
Document the direct Node command.
- Test manifest-listed evidence, rejection of unlisted files and encoded
traversal, cache reuse, and offline misses with a real loopback server
and a stub downloader.

## Verification

- Focused report, history, and preview tests: 75 passed in 6 files.
- Report Playwright spec: 4 passed in Chrome with a standalone config
that does not start the application server.
- `node scripts/check-maps.mjs`: passed.
- `git diff --check`: passed.
- Manually checked retained campaign reports, feature filters, folding
indexes, failure links, card details, gallery links, mobile layout, and
keyboard navigation.
- `pnpm build`: passed.
- `pnpm -r typecheck`: passed.
- `pnpm test:e2e:runner:typecheck`: passed.
- `pnpm test:run`: stopped after 15 native session database cases
failed. The local Postgres package first lacked dynamic-library links.
After link hydration, the same cases still failed. A direct `initdb`
check then reported `could not create semaphores: No space left on
device` and identified the macOS semaphore limit. The complete local
suite has not passed.
- `pnpm test:e2e:runner:unit`: 3,250 passed, 6 failed, 2 skipped. All
six failures reproduced against unchanged `origin/master` report sources
with the same installed dependencies. They concern Copilot admission and
its Daytona image input, plus the launcher test that runs those
assertions. No report or history test failed.
- No live provider evaluation was run for this presentation change.
- CI on `b1a9c06823` first failed browser shard 3/8. The agent-chat
split-handoff test reached cleanup, where cancelling a run returned HTTP
500 from a Postgres deadlock. Cleanup did not restore the experiment
setting, and a later test failed its initial setting check. This shard
does not contain the report spec. One targeted rerun passed, and that
full workflow then passed. This shows recovery; no production fix was
made for the deadlock. The later preview-test commit requires its own
fresh checks.

To review the design, run `node --watch --import
./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/preview.ts`.
Open the history address printed by the command, then open a campaign.
The README covers offline mode and cache refresh.

## Risks

- Feature associations use the current map. They describe linked
scenarios, not proof that a whole capability is covered. Historical
grades remain unchanged.
- New or unfamiliar cases remain unmapped until their assertions are
inspected.
- The preview needs the public history host on first access. Offline
mode requires cached files. It binds to the loopback interface and
serves manifest-listed public evidence.
- Hiding routine diagnostics adds one disclosure step. Failed or
incomplete outcomes remain visible.

This is report polish. It does not duplicate planned core work in
`ROADMAP.md`.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, code execution,
and browser automation. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 14:02:19 -05:00

86 lines
5.6 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import type { ReportExecution } from "./report-catalog.js";
import type { RunnerE2ESuiteSummary } from "./types.js";
function html(value: unknown) {
return String(value ?? "").replace(/[&<>"']/g, (character) =>
({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[character]!,
);
}
function uniqueCases(catalog: readonly ReportExecution[]) {
const cases = new Map<string, ReportExecution>();
for (const execution of catalog) {
const key = `${execution.suite.id}.${execution.task.id}`;
if (!cases.has(key)) cases.set(key, execution);
}
return [...cases.values()];
}
export function renderSuiteCaseIndex(catalog: readonly ReportExecution[]) {
return `<ul class="suite-case-list">${uniqueCases(catalog).map((execution) => `<li>
<a href="#execution-${html(execution.id)}" data-report-case-link="${html(`${execution.suite.id}.${execution.task.id}`)}"><span class="case-status-dot" aria-hidden="true"></span><span>${html(execution.task.label)}</span></a>
</li>`).join("")}</ul>`;
}
export function renderCampaignSuiteSummary(
suites: readonly RunnerE2ESuiteSummary[],
catalog: readonly ReportExecution[],
reportHref = "",
showCases = false,
) {
const labels = new Map(catalog.map((execution) => [execution.suite.id, execution.suite.label]));
const selected = suites.filter((suite) => suite.selected > 0);
if (selected.length === 0) return "";
return `<ul class="campaign-suites" aria-label="Suite results">${selected.map((suite) => {
const passed = suite.passed === suite.selected && suite.cleanupPassed;
const label = labels.get(suite.suiteId) ?? suite.suiteId;
const suiteCatalog = catalog.filter((execution) => execution.suite.id === suite.suiteId);
const caseCount = uniqueCases(suiteCatalog).length;
const fold = showCases && caseCount > 0;
const heading = `<span class="campaign-suite-title"><a href="${html(reportHref)}#suite-${html(suite.suiteId)}">${html(label)}</a>${fold ? `<small>${caseCount} ${caseCount === 1 ? "case" : "cases"}</small>` : ""}</span>
<strong>${suite.passed}/${suite.selected}</strong>
<span class="campaign-suite-state ${passed ? "campaign-suite-passed" : "campaign-suite-attention"}">${passed ? "Passed" : "Needs attention"}${suite.incomplete ? ` · ${suite.incomplete} incomplete` : ""}</span>`;
return `<li>${fold ? `<details class="campaign-suite-details">
<summary class="campaign-suite-row">${heading}</summary>
<nav class="campaign-case-index" aria-label="${html(label)} cases">${renderSuiteCaseIndex(suiteCatalog)}</nav>
</details>` : `<div class="campaign-suite-row">${heading}</div>`}
</li>`;
}).join("")}</ul>`;
}
export const campaignSuiteSummaryStyles = `
.campaign-suites { display: grid; margin: 0 0 32px; padding: 0; list-style: none; }
.campaign-suites > li { border-bottom: 1px solid var(--border); }
.campaign-suite-row { display: grid; grid-template-columns: minmax(0, 1fr) auto 150px; align-items: baseline; gap: 24px; padding: 12px 0; }
.campaign-suite-details > summary { grid-template-columns: 16px minmax(0, 1fr) auto 150px; gap: 12px; cursor: pointer; list-style: none; }
.campaign-suite-details > summary::-webkit-details-marker { display: none; }
.campaign-suite-details > summary::before { content: "›"; align-self: center; color: var(--muted-foreground, var(--muted)); font-size: 20px; line-height: 1; text-align: center; }
.campaign-suite-details[open] > summary::before { transform: rotate(90deg); }
.campaign-suite-title { display: flex; flex-wrap: wrap; align-items: baseline; gap: 4px 12px; min-width: 0; }
.campaign-suite-title small { color: var(--muted-foreground, var(--muted)); font-size: 11px; }
.campaign-case-index { padding: 0 0 16px 28px; }
.campaign-suites a { width: fit-content; font-weight: 500; text-decoration: none; }
.campaign-suites a:hover { text-decoration: underline; }
.campaign-suites strong { font-family: var(--font-mono, var(--mono)); font-variant-numeric: tabular-nums; font-weight: 600; }
.campaign-suite-state { font-size: 12px; text-align: right; }
.campaign-suite-passed { color: var(--pass-text, var(--pass)); }
.campaign-suite-attention { color: var(--fail-text, var(--fail)); }
.suite-case-list { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 6px 24px; margin: 0; padding: 0; list-style: none; }
.suite-case-list a { display: inline-flex; align-items: baseline; gap: 8px; padding: 4px 0; color: var(--muted-foreground, var(--muted)); font-size: 12px; font-weight: 400; text-decoration: none; text-underline-offset: 3px; overflow-wrap: anywhere; }
.suite-case-list a:hover { color: var(--foreground); text-decoration: underline; }
.case-status-dot { display: inline-block; flex: 0 0 7px; width: 7px; height: 7px; border-radius: 50%; background: var(--muted-foreground, var(--muted)); }
[data-case-status="passed"] .case-status-dot { background: var(--pass-text, var(--pass)); }
[data-case-status="failed"] .case-status-dot { background: var(--fail-text, var(--fail)); }
[data-case-status="incomplete"] .case-status-dot, [data-case-status="missing"] .case-status-dot { background: var(--missing-text); }
@media (max-width: 980px) {
.suite-case-list { grid-template-columns: repeat(2, minmax(0, 1fr)); }
}
@media (max-width: 520px) {
.campaign-suite-row { grid-template-columns: minmax(0, 1fr) auto; gap: 4px 16px; }
.campaign-suite-state { grid-column: 1 / -1; text-align: left; }
.campaign-suite-details > summary { grid-template-columns: 16px minmax(0, 1fr) auto; gap: 4px 12px; }
.campaign-suite-details .campaign-suite-state { grid-column: 2 / -1; }
.suite-case-list { grid-template-columns: 1fr; gap: 2px; }
}
`;