Files
PaperClipAI/tests/runner-e2e/feature-coverage.ts
T
DottaandPaperclip 2b16262280 Improve runner E2E report navigation and local previews (#16996)
## Thinking Path

> - Paperclip is the open source app that people use to manage AI agents
for work.
> - Product E2E campaigns check user journeys across agent profiles and
environments.
> - Readers use the campaign history and reports to inspect results and
retained evidence.
> - The reports showed image summaries, many unselected suites, and a
long sticky index.
> - Individual case cards also showed routine diagnostics before the
visual evidence.
> - This pull request adds text summaries, feature filters, folding case
indexes, and simpler cards.
> - A local preview uses retained public data so designers can change
reports without another paid campaign.

## Linked Issues or Issue Description

**Current behavior**

The report is hard to scan as the journey catalog grows. Suite links can
point into hidden results. Case indexes do not show whether a case
passed across its configurations. Design changes also need a realistic
local preview.

**Proposed behavior**

Show the selected campaign as text. Link journeys to current feature-map
capabilities. Add folding case indexes with status indicators, reliable
section links, and a back-to-top control. Keep failed outcomes visible
and put routine diagnostics under Test details. Render retained public
campaign data with local templates.

**Reason and benefit**

Readers can find a feature or failed case faster. Designers can iterate
against real retained data without starting an application instance or
using provider credentials.

**Breaking changes**

No result schema or grading change. The report stops embedding the
generated summary PNG. The publisher still produces that artifact for
existing consumers.

Related work: #12889 added the existing matrix browser. #12846 concerns
the public image publication boundary. This PR changes report
presentation and local preview behavior.

## What Changed

- Replace image summaries with shared HTML suite summaries on the
history and campaign pages.
- Show selected executions and preserve missing results for selected
cells.
- Add explicit journey-to-capability associations, feature filters, and
a coverage view with unmapped journeys.
- Add folding case indexes, section case lists, and red, green, or amber
status indicators across matching configurations.
- Fix anchor navigation through filters and add a floating back-to-top
control.
- Simplify case cards. Keep failure reasons visible. Fold routine
checks, billing, and context into Test details.
- Add an original-image link to the gallery.
- Add a local preview utility with public data caching and offline mode.
Document the direct Node command.
- Test manifest-listed evidence, rejection of unlisted files and encoded
traversal, cache reuse, and offline misses with a real loopback server
and a stub downloader.

## Verification

- Focused report, history, and preview tests: 75 passed in 6 files.
- Report Playwright spec: 4 passed in Chrome with a standalone config
that does not start the application server.
- `node scripts/check-maps.mjs`: passed.
- `git diff --check`: passed.
- Manually checked retained campaign reports, feature filters, folding
indexes, failure links, card details, gallery links, mobile layout, and
keyboard navigation.
- `pnpm build`: passed.
- `pnpm -r typecheck`: passed.
- `pnpm test:e2e:runner:typecheck`: passed.
- `pnpm test:run`: stopped after 15 native session database cases
failed. The local Postgres package first lacked dynamic-library links.
After link hydration, the same cases still failed. A direct `initdb`
check then reported `could not create semaphores: No space left on
device` and identified the macOS semaphore limit. The complete local
suite has not passed.
- `pnpm test:e2e:runner:unit`: 3,250 passed, 6 failed, 2 skipped. All
six failures reproduced against unchanged `origin/master` report sources
with the same installed dependencies. They concern Copilot admission and
its Daytona image input, plus the launcher test that runs those
assertions. No report or history test failed.
- No live provider evaluation was run for this presentation change.
- CI on `b1a9c06823` first failed browser shard 3/8. The agent-chat
split-handoff test reached cleanup, where cancelling a run returned HTTP
500 from a Postgres deadlock. Cleanup did not restore the experiment
setting, and a later test failed its initial setting check. This shard
does not contain the report spec. One targeted rerun passed, and that
full workflow then passed. This shows recovery; no production fix was
made for the deadlock. The later preview-test commit requires its own
fresh checks.

To review the design, run `node --watch --import
./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/preview.ts`.
Open the history address printed by the command, then open a campaign.
The README covers offline mode and cache refresh.

## Risks

- Feature associations use the current map. They describe linked
scenarios, not proof that a whole capability is covered. Historical
grades remain unchanged.
- New or unfamiliar cases remain unmapped until their assertions are
inspected.
- The preview needs the public history host on first access. Offline
mode requires cached files. It binds to the loopback interface and
serves manifest-listed public evidence.
- Hiding routine diagnostics adds one disclosure step. Failed or
incomplete outcomes remain visible.

This is report polish. It does not duplicate planned core work in
`ROADMAP.md`.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, code execution,
and browser automation. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 14:02:19 -05:00

64 lines
4.9 KiB
TypeScript

import { journeyCapabilities, journeyFeatureIds, journeyFeatureMap, journeyFeatures } from "./journey-features.js";
import type { ReportExecution } from "./report-catalog.js";
function html(value: unknown) {
return String(value ?? "").replace(/[&<>"']/g, (character) =>
({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[character]!,
);
}
export function renderJourneyFeatureOptions(catalog: readonly ReportExecution[]) {
const selected = new Set(catalog.flatMap(journeyFeatureIds));
return journeyFeatureMap.sections.map((section) => {
const features = section.features.filter((feature) => selected.has(feature.id));
return features.length ? `<optgroup label="${html(section.title)}">${features.map((feature) =>
`<option value="${html(feature.id)}">${html(feature.title)}</option>`,
).join("")}</optgroup>` : "";
}).join("") + (catalog.some((execution) => journeyCapabilities(execution).length === 0)
? '<option value="__unmapped__">Unmapped journeys</option>' : "");
}
export function renderJourneyFeatureCoverage(
catalog: readonly ReportExecution[],
entries: ReadonlyMap<string, { valid: boolean }>,
) {
const linked = new Set(catalog.flatMap((execution) => journeyCapabilities(execution).map((capability) => capability.id)));
const mappedTests = catalog.filter((execution) => journeyCapabilities(execution).length > 0).length;
const totalCapabilities = journeyFeatures.reduce((total, feature) => total + feature.capabilities.length, 0);
return `<details class="feature-coverage" id="feature-coverage">
<summary>Feature coverage <span>${linked.size} of ${totalCapabilities} capabilities linked · ${mappedTests} of ${catalog.length} tests mapped</span></summary>
<p>Associations use the current feature map. Counts describe linked test scenarios; a passing journey does not establish full coverage of a capability.</p>
${mappedTests < catalog.length ? `<button type="button" class="feature-choice" data-report-feature-choice="__unmapped__">Find ${catalog.length - mappedTests} unmapped tests</button>` : ""}
<div class="feature-coverage-sections">${journeyFeatureMap.sections.map((section) => `<section>
<h3>${html(section.title)}</h3>
<table><thead><tr><th>Feature</th><th>Capabilities linked</th><th>Linked test results</th></tr></thead><tbody>${section.features.map((feature) => {
const tests = catalog.filter((execution) => journeyFeatureIds(execution).includes(feature.id));
const mapped = feature.capabilities.filter((capability) => linked.has(capability.id));
const passed = tests.filter((execution) => entries.get(execution.id)?.valid).length;
const missing = tests.filter((execution) => !entries.has(execution.id)).length;
return `<tr><th scope="row">${tests.length
? `<button type="button" class="feature-choice" data-report-feature-choice="${html(feature.id)}">${html(feature.title)}</button>`
: html(feature.title)}<small>${mapped.map((capability) => html(capability.label)).join(" · ")}</small></th>
<td>${mapped.length}/${feature.capabilities.length}</td>
<td>${tests.length ? `${passed}/${tests.length} passed${tests.length - passed - missing ? ` · ${tests.length - passed - missing} failed or incomplete` : ""}${missing ? ` · ${missing} missing` : ""}` : "No mapped journey"}</td></tr>`;
}).join("")}</tbody></table>
</section>`).join("")}</div>
</details>`;
}
export const journeyFeatureStyles = `
.feature-coverage { margin: 0 0 24px; padding: 14px 0; border-top: 1px solid var(--border); border-bottom: 1px solid var(--border); }
.feature-coverage > summary { cursor: pointer; font-weight: 600; }
.feature-coverage > summary span { margin-left: 12px; color: var(--muted-foreground); font-size: 12px; font-weight: 400; }
.feature-coverage > p { max-width: 780px; color: var(--muted-foreground); font-size: 12px; }
.feature-coverage-sections { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 24px; margin-top: 24px; }
.feature-coverage h3 { margin: 0 0 12px; font-size: 14px; }
.feature-coverage table { width: 100%; border-collapse: collapse; font-size: 11px; }
.feature-coverage th, .feature-coverage td { padding: 10px 8px; border-bottom: 1px solid var(--border); text-align: left; vertical-align: top; }
.feature-coverage thead th { color: var(--muted-foreground); font-weight: 500; }
.feature-coverage th small { display: block; margin-top: 4px; color: var(--muted-foreground); font-weight: 400; }
.feature-choice { padding: 0; border: 0; background: transparent; color: inherit; cursor: pointer; font: inherit; text-align: left; text-decoration: underline; text-underline-offset: 3px; }
.case-features { display: flex; flex-wrap: wrap; gap: 4px 12px; margin: 8px 0; color: var(--muted-foreground); font-size: 10px; }
@media (max-width: 860px) { .feature-coverage-sections { grid-template-columns: 1fr; } .feature-coverage > summary span { display: block; margin: 4px 0 0; } }
`;