mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 16:27:18 +02:00
## Thinking Path > - Paperclip is the open source app that people use to manage AI agents for work. > - Product E2E campaigns check user journeys across agent profiles and environments. > - Readers use the campaign history and reports to inspect results and retained evidence. > - The reports showed image summaries, many unselected suites, and a long sticky index. > - Individual case cards also showed routine diagnostics before the visual evidence. > - This pull request adds text summaries, feature filters, folding case indexes, and simpler cards. > - A local preview uses retained public data so designers can change reports without another paid campaign. ## Linked Issues or Issue Description **Current behavior** The report is hard to scan as the journey catalog grows. Suite links can point into hidden results. Case indexes do not show whether a case passed across its configurations. Design changes also need a realistic local preview. **Proposed behavior** Show the selected campaign as text. Link journeys to current feature-map capabilities. Add folding case indexes with status indicators, reliable section links, and a back-to-top control. Keep failed outcomes visible and put routine diagnostics under Test details. Render retained public campaign data with local templates. **Reason and benefit** Readers can find a feature or failed case faster. Designers can iterate against real retained data without starting an application instance or using provider credentials. **Breaking changes** No result schema or grading change. The report stops embedding the generated summary PNG. The publisher still produces that artifact for existing consumers. Related work: #12889 added the existing matrix browser. #12846 concerns the public image publication boundary. This PR changes report presentation and local preview behavior. ## What Changed - Replace image summaries with shared HTML suite summaries on the history and campaign pages. - Show selected executions and preserve missing results for selected cells. - Add explicit journey-to-capability associations, feature filters, and a coverage view with unmapped journeys. - Add folding case indexes, section case lists, and red, green, or amber status indicators across matching configurations. - Fix anchor navigation through filters and add a floating back-to-top control. - Simplify case cards. Keep failure reasons visible. Fold routine checks, billing, and context into Test details. - Add an original-image link to the gallery. - Add a local preview utility with public data caching and offline mode. Document the direct Node command. - Test manifest-listed evidence, rejection of unlisted files and encoded traversal, cache reuse, and offline misses with a real loopback server and a stub downloader. ## Verification - Focused report, history, and preview tests: 75 passed in 6 files. - Report Playwright spec: 4 passed in Chrome with a standalone config that does not start the application server. - `node scripts/check-maps.mjs`: passed. - `git diff --check`: passed. - Manually checked retained campaign reports, feature filters, folding indexes, failure links, card details, gallery links, mobile layout, and keyboard navigation. - `pnpm build`: passed. - `pnpm -r typecheck`: passed. - `pnpm test:e2e:runner:typecheck`: passed. - `pnpm test:run`: stopped after 15 native session database cases failed. The local Postgres package first lacked dynamic-library links. After link hydration, the same cases still failed. A direct `initdb` check then reported `could not create semaphores: No space left on device` and identified the macOS semaphore limit. The complete local suite has not passed. - `pnpm test:e2e:runner:unit`: 3,250 passed, 6 failed, 2 skipped. All six failures reproduced against unchanged `origin/master` report sources with the same installed dependencies. They concern Copilot admission and its Daytona image input, plus the launcher test that runs those assertions. No report or history test failed. - No live provider evaluation was run for this presentation change. - CI on `b1a9c06823` first failed browser shard 3/8. The agent-chat split-handoff test reached cleanup, where cancelling a run returned HTTP 500 from a Postgres deadlock. Cleanup did not restore the experiment setting, and a later test failed its initial setting check. This shard does not contain the report spec. One targeted rerun passed, and that full workflow then passed. This shows recovery; no production fix was made for the deadlock. The later preview-test commit requires its own fresh checks. To review the design, run `node --watch --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/preview.ts`. Open the history address printed by the command, then open a campaign. The README covers offline mode and cache refresh. ## Risks - Feature associations use the current map. They describe linked scenarios, not proof that a whole capability is covered. Historical grades remain unchanged. - New or unfamiliar cases remain unmapped until their assertions are inspected. - The preview needs the public history host on first access. Offline mode requires cached files. It binds to the loopback interface and serves manifest-listed public evidence. - Hiding routine diagnostics adds one disclosure step. Failed or incomplete outcomes remain visible. This is report polish. It does not duplicate planned core work in `ROADMAP.md`. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, code execution, and browser automation. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
64 lines
4.9 KiB
TypeScript
64 lines
4.9 KiB
TypeScript
import { journeyCapabilities, journeyFeatureIds, journeyFeatureMap, journeyFeatures } from "./journey-features.js";
|
|
import type { ReportExecution } from "./report-catalog.js";
|
|
|
|
function html(value: unknown) {
|
|
return String(value ?? "").replace(/[&<>"']/g, (character) =>
|
|
({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" })[character]!,
|
|
);
|
|
}
|
|
|
|
export function renderJourneyFeatureOptions(catalog: readonly ReportExecution[]) {
|
|
const selected = new Set(catalog.flatMap(journeyFeatureIds));
|
|
return journeyFeatureMap.sections.map((section) => {
|
|
const features = section.features.filter((feature) => selected.has(feature.id));
|
|
return features.length ? `<optgroup label="${html(section.title)}">${features.map((feature) =>
|
|
`<option value="${html(feature.id)}">${html(feature.title)}</option>`,
|
|
).join("")}</optgroup>` : "";
|
|
}).join("") + (catalog.some((execution) => journeyCapabilities(execution).length === 0)
|
|
? '<option value="__unmapped__">Unmapped journeys</option>' : "");
|
|
}
|
|
|
|
export function renderJourneyFeatureCoverage(
|
|
catalog: readonly ReportExecution[],
|
|
entries: ReadonlyMap<string, { valid: boolean }>,
|
|
) {
|
|
const linked = new Set(catalog.flatMap((execution) => journeyCapabilities(execution).map((capability) => capability.id)));
|
|
const mappedTests = catalog.filter((execution) => journeyCapabilities(execution).length > 0).length;
|
|
const totalCapabilities = journeyFeatures.reduce((total, feature) => total + feature.capabilities.length, 0);
|
|
return `<details class="feature-coverage" id="feature-coverage">
|
|
<summary>Feature coverage <span>${linked.size} of ${totalCapabilities} capabilities linked · ${mappedTests} of ${catalog.length} tests mapped</span></summary>
|
|
<p>Associations use the current feature map. Counts describe linked test scenarios; a passing journey does not establish full coverage of a capability.</p>
|
|
${mappedTests < catalog.length ? `<button type="button" class="feature-choice" data-report-feature-choice="__unmapped__">Find ${catalog.length - mappedTests} unmapped tests</button>` : ""}
|
|
<div class="feature-coverage-sections">${journeyFeatureMap.sections.map((section) => `<section>
|
|
<h3>${html(section.title)}</h3>
|
|
<table><thead><tr><th>Feature</th><th>Capabilities linked</th><th>Linked test results</th></tr></thead><tbody>${section.features.map((feature) => {
|
|
const tests = catalog.filter((execution) => journeyFeatureIds(execution).includes(feature.id));
|
|
const mapped = feature.capabilities.filter((capability) => linked.has(capability.id));
|
|
const passed = tests.filter((execution) => entries.get(execution.id)?.valid).length;
|
|
const missing = tests.filter((execution) => !entries.has(execution.id)).length;
|
|
return `<tr><th scope="row">${tests.length
|
|
? `<button type="button" class="feature-choice" data-report-feature-choice="${html(feature.id)}">${html(feature.title)}</button>`
|
|
: html(feature.title)}<small>${mapped.map((capability) => html(capability.label)).join(" · ")}</small></th>
|
|
<td>${mapped.length}/${feature.capabilities.length}</td>
|
|
<td>${tests.length ? `${passed}/${tests.length} passed${tests.length - passed - missing ? ` · ${tests.length - passed - missing} failed or incomplete` : ""}${missing ? ` · ${missing} missing` : ""}` : "No mapped journey"}</td></tr>`;
|
|
}).join("")}</tbody></table>
|
|
</section>`).join("")}</div>
|
|
</details>`;
|
|
}
|
|
|
|
export const journeyFeatureStyles = `
|
|
.feature-coverage { margin: 0 0 24px; padding: 14px 0; border-top: 1px solid var(--border); border-bottom: 1px solid var(--border); }
|
|
.feature-coverage > summary { cursor: pointer; font-weight: 600; }
|
|
.feature-coverage > summary span { margin-left: 12px; color: var(--muted-foreground); font-size: 12px; font-weight: 400; }
|
|
.feature-coverage > p { max-width: 780px; color: var(--muted-foreground); font-size: 12px; }
|
|
.feature-coverage-sections { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 24px; margin-top: 24px; }
|
|
.feature-coverage h3 { margin: 0 0 12px; font-size: 14px; }
|
|
.feature-coverage table { width: 100%; border-collapse: collapse; font-size: 11px; }
|
|
.feature-coverage th, .feature-coverage td { padding: 10px 8px; border-bottom: 1px solid var(--border); text-align: left; vertical-align: top; }
|
|
.feature-coverage thead th { color: var(--muted-foreground); font-weight: 500; }
|
|
.feature-coverage th small { display: block; margin-top: 4px; color: var(--muted-foreground); font-weight: 400; }
|
|
.feature-choice { padding: 0; border: 0; background: transparent; color: inherit; cursor: pointer; font: inherit; text-align: left; text-decoration: underline; text-underline-offset: 3px; }
|
|
.case-features { display: flex; flex-wrap: wrap; gap: 4px 12px; margin: 8px 0; color: var(--muted-foreground); font-size: 10px; }
|
|
@media (max-width: 860px) { .feature-coverage-sections { grid-template-columns: 1fr; } .feature-coverage > summary span { display: block; margin: 4px 0 0; } }
|
|
`;
|