PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-23 17:33:51 +02:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	f89c37791c	Add campaign sidebar and case navigation to runner E2E reports (#17010 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E campaigns test runner behavior across agent profiles and environments. > - The retained reports contain many suites, cases, and execution previews. > - A long report makes it hard to find a case or return to the campaign context. > - This pull request adds campaign navigation, case breadcrumbs, and separate global history pages. > - Users can scan all execution previews or inspect one case without losing their place. ## Linked Issues or Issue Description What existing behavior does this improve? Navigation in the retained Runner E2E campaign reports and history homepage. Subsystem affected Test report infrastructure in `tests/runner-e2e/`. Current behavior Reports show a long configuration matrix. The suite index and campaign history share the same page. Case links do not provide a focused inspection view. Proposed behavior Global overview, history, and attention pages show global navigation. A specific campaign adds a folding suite tree. All results keeps the full scrollable report. A case opens with breadcrumbs and an execution picker. Reason and benefit Users can find a case, compare its executions, and return to their reading position. Global pages do not show suites from an unrelated campaign. Breaking changes The report HTML layout changes. Retained result schemas, execution identities, and suite and execution anchors remain unchanged. Additional context Refs: #16996. This extends the merged report navigation and local preview work. The duplicate PR and issue search found no open PR for this change. ## What Changed - Add shared report navigation with the Paperclip brand mark and wordmark. - Scope the suite tree to the current campaign. Keep global overview, history, and attention pages separate. - Group all execution previews by suite and case. Reuse the existing evidence cards and image gallery. - Add focused case inspection, configuration selection, breadcrumbs, filter handling, and scroll restoration. - Move historical trends to the global history page. Keep billing details, missing results, and failure evidence available. - Update report documentation and browser regressions for filters, status dots, configuration reloads, breadcrumbs, and scroll restoration. ## Verification - [Final CI](https://github.com/paperclipai/paperclip/actions/runs/39500401411) passes on `89a8b5c290de3cfb9bdc061be3c88a29fd160a62`. Typecheck, build, runner checks, server suites, and all browser shards pass. - All four report browser regressions pass. They cover filters, gallery navigation, repeated suite jumps, the folding sidebar, breadcrumbs, scroll restoration, configuration selection and reload, mobile navigation, and aggregate case status. - The 63 report, history, preview-boundary, and navigation tests pass. Runner E2E typecheck passes with `node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json`. - Manual browser checks covered desktop and a 390-pixel viewport. Retained campaigns covered 209 executions in eight suites, 179 executions in seven suites, and 15 executions in one suite. Browser fixture checks also confirmed filtered inspection and aggregate status. No paid evaluations ran. - The initial report browser tests still targeted the old layout and routes. They were updated for the new navigation. Review also exposed filtered-inspection and aggregate-status defects; both were fixed before the final passing browser run. - The unchanged Cases UI grouping test [failed once on the final head](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118898559197). Its full file also failed locally, while the isolated test passed. The same UI code passed in earlier CI. A [single CI retry](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118903333430) passed all 3,978 UI tests and 286 CLI tests without code changes. The cause remains unresolved. - Two jobs on an intermediate head were interrupted by runner shutdown signals. Those remain interrupted results. The complete final-head CI run supersedes them for merge verification. - Workspace `pnpm -r typecheck` and `pnpm build` also passed locally before the rebase. The full local `pnpm test:run` attempt encountered repeated embedded PostgreSQL startup failures and was stopped. It is not a passing local full-suite result; final CI provides the full verification. ## Risks - Navigation and hash handling can select the wrong case or hide results. Focused tests and browser checks cover these paths. - Historical campaign links must resolve from both the preview and published report locations. - The report layout changes. Graders, prompts, retained outcomes, and billing calculations do not change. ## Model Used - OpenAI Codex with `gpt-6-astra` and `gpt-6.1-sol`. Used reasoning, repository tools, code execution, and browser verification. The runtime did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 16:50:53 -05:00
DottaandPaperclip	2b16262280	Improve runner E2E report navigation and local previews (#16996 ) ## Thinking Path > - Paperclip is the open source app that people use to manage AI agents for work. > - Product E2E campaigns check user journeys across agent profiles and environments. > - Readers use the campaign history and reports to inspect results and retained evidence. > - The reports showed image summaries, many unselected suites, and a long sticky index. > - Individual case cards also showed routine diagnostics before the visual evidence. > - This pull request adds text summaries, feature filters, folding case indexes, and simpler cards. > - A local preview uses retained public data so designers can change reports without another paid campaign. ## Linked Issues or Issue Description Current behavior The report is hard to scan as the journey catalog grows. Suite links can point into hidden results. Case indexes do not show whether a case passed across its configurations. Design changes also need a realistic local preview. Proposed behavior Show the selected campaign as text. Link journeys to current feature-map capabilities. Add folding case indexes with status indicators, reliable section links, and a back-to-top control. Keep failed outcomes visible and put routine diagnostics under Test details. Render retained public campaign data with local templates. Reason and benefit Readers can find a feature or failed case faster. Designers can iterate against real retained data without starting an application instance or using provider credentials. Breaking changes No result schema or grading change. The report stops embedding the generated summary PNG. The publisher still produces that artifact for existing consumers. Related work: #12889 added the existing matrix browser. #12846 concerns the public image publication boundary. This PR changes report presentation and local preview behavior. ## What Changed - Replace image summaries with shared HTML suite summaries on the history and campaign pages. - Show selected executions and preserve missing results for selected cells. - Add explicit journey-to-capability associations, feature filters, and a coverage view with unmapped journeys. - Add folding case indexes, section case lists, and red, green, or amber status indicators across matching configurations. - Fix anchor navigation through filters and add a floating back-to-top control. - Simplify case cards. Keep failure reasons visible. Fold routine checks, billing, and context into Test details. - Add an original-image link to the gallery. - Add a local preview utility with public data caching and offline mode. Document the direct Node command. - Test manifest-listed evidence, rejection of unlisted files and encoded traversal, cache reuse, and offline misses with a real loopback server and a stub downloader. ## Verification - Focused report, history, and preview tests: 75 passed in 6 files. - Report Playwright spec: 4 passed in Chrome with a standalone config that does not start the application server. - `node scripts/check-maps.mjs`: passed. - `git diff --check`: passed. - Manually checked retained campaign reports, feature filters, folding indexes, failure links, card details, gallery links, mobile layout, and keyboard navigation. - `pnpm build`: passed. - `pnpm -r typecheck`: passed. - `pnpm test:e2e:runner:typecheck`: passed. - `pnpm test:run`: stopped after 15 native session database cases failed. The local Postgres package first lacked dynamic-library links. After link hydration, the same cases still failed. A direct `initdb` check then reported `could not create semaphores: No space left on device` and identified the macOS semaphore limit. The complete local suite has not passed. - `pnpm test:e2e:runner:unit`: 3,250 passed, 6 failed, 2 skipped. All six failures reproduced against unchanged `origin/master` report sources with the same installed dependencies. They concern Copilot admission and its Daytona image input, plus the launcher test that runs those assertions. No report or history test failed. - No live provider evaluation was run for this presentation change. - CI on `b1a9c06823` first failed browser shard 3/8. The agent-chat split-handoff test reached cleanup, where cancelling a run returned HTTP 500 from a Postgres deadlock. Cleanup did not restore the experiment setting, and a later test failed its initial setting check. This shard does not contain the report spec. One targeted rerun passed, and that full workflow then passed. This shows recovery; no production fix was made for the deadlock. The later preview-test commit requires its own fresh checks. To review the design, run `node --watch --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/preview.ts`. Open the history address printed by the command, then open a campaign. The README covers offline mode and cache refresh. ## Risks - Feature associations use the current map. They describe linked scenarios, not proof that a whole capability is covered. Historical grades remain unchanged. - New or unfamiliar cases remain unmapped until their assertions are inspected. - The preview needs the public history host on first access. Offline mode requires cached files. It binds to the loopback interface and serves manifest-listed public evidence. - Hiding routine diagnostics adds one disclosure step. Failed or incomplete outcomes remain visible. This is report polish. It does not duplicate planned core work in `ROADMAP.md`. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, code execution, and browser automation. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 14:02:19 -05:00
DottaandPaperclip	4a7172f5ac	feat(runner-e2e): improve matrix report browsing (#12889 ) ## Thinking Path > - Paperclip is the open source app that people use to manage AI agents for work. > - The runner E2E system verifies agent profiles across supported execution environments. > - Its generated report is the main surface for inspecting those results and their visual evidence. > - Expanded matcher content could change the matrix column widths and make comparisons difficult. > - Screenshots also required extra navigation, and the report did not have current-result search and filters. > - This pull request stabilizes the matrix layout and makes visual evidence directly browsable. > - The benefit is faster inspection of retained test evidence without another paid matrix run. ## Linked Issues or Issue Description What happened? The runner E2E report changed matrix column widths when a matcher table expanded. The report also made screenshot comparison and current-result discovery slower than necessary. Expected behavior The matrix columns must remain stable. Each retained screenshot must appear as a thumbnail. The gallery must support keyboard navigation and show the relevant execution metadata. The report must support client-side search and filters. Steps to reproduce 1. Open a runner E2E matrix report that contains retained screenshots. 2. Expand the matcher details in a matrix cell. 3. Observe the matrix column movement in the old report. Paperclip version or commit `8430bd897` Deployment mode Generated static runner E2E report. ## What Changed - Keep matrix and matcher table widths stable when details expand. - Show retained screenshot thumbnails in each test card. - Add a full-screen evidence gallery with mouse, keyboard, and swipe navigation. - Show agent, environment, runtime, status, duration, token, and matcher data in the gallery header. - Add client-side search and profile, environment, suite, and status filters below the report section tabs. - Keep the filters in normal document flow while the report tabs remain sticky. - Add report generator assertions for the new layout and controls. - Add Playwright coverage for filtering, stable matcher expansion, and filtered gallery navigation. ## Verification - `pnpm test:e2e:runner:typecheck` - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/report.test.ts` - `PAPERCLIP_PLAYWRIGHT_CHANNEL=chrome pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/runner-e2e-dashboard.spec.ts` - `node --test ./scripts/__tests__/e2e-shard.test.mjs` - `pnpm check:token-gates` - `pnpm -r typecheck` - `pnpm test:run` - `pnpm build` - Regenerated the report from GitHub Actions run `33963318820` without rerunning the matrix. - Verified 121 retained thumbnails across 66 results in a local browser. - Verified that expanded matchers keep matrix widths at 260, 486, and 486 pixels in a 1280-pixel viewport. - Verified search, filters, modal metadata, and arrow-key gallery navigation. ## Risks Low risk. This change only modifies the static runner E2E report generator and its tests. It does not change runner execution or retained evidence data. > This work is focused report polish. It does not duplicate planned core work in `ROADMAP.md`. ## Model Used OpenAI Codex with `gpt-5.6-sol`. Codex desktop managed the context window. The model used reasoning, filesystem tools, code execution, GitHub CLI access, and in-app browser verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-05 07:46:05 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

f89c37791c

Add campaign sidebar and case navigation to runner E2E reports (#17010 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E campaigns test runner behavior across agent profiles and
environments.
> - The retained reports contain many suites, cases, and execution
previews.
> - A long report makes it hard to find a case or return to the campaign
context.
> - This pull request adds campaign navigation, case breadcrumbs, and
separate global history pages.
> - Users can scan all execution previews or inspect one case without
losing their place.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Navigation in the retained Runner E2E campaign reports and history
homepage.

**Subsystem affected**

Test report infrastructure in `tests/runner-e2e/`.

**Current behavior**

Reports show a long configuration matrix. The suite index and campaign
history share the same page. Case links do not provide a focused
inspection view.

**Proposed behavior**

Global overview, history, and attention pages show global navigation. A
specific campaign adds a folding suite tree. All results keeps the full
scrollable report. A case opens with breadcrumbs and an execution
picker.

**Reason and benefit**

Users can find a case, compare its executions, and return to their
reading position. Global pages do not show suites from an unrelated
campaign.

**Breaking changes**

The report HTML layout changes. Retained result schemas, execution
identities, and suite and execution anchors remain unchanged.

**Additional context**

Refs: #16996. This extends the merged report navigation and local
preview work. The duplicate PR and issue search found no open PR for
this change.

## What Changed

- Add shared report navigation with the Paperclip brand mark and
wordmark.
- Scope the suite tree to the current campaign. Keep global overview,
history, and attention pages separate.
- Group all execution previews by suite and case. Reuse the existing
evidence cards and image gallery.
- Add focused case inspection, configuration selection, breadcrumbs,
filter handling, and scroll restoration.
- Move historical trends to the global history page. Keep billing
details, missing results, and failure evidence available.
- Update report documentation and browser regressions for filters,
status dots, configuration reloads, breadcrumbs, and scroll restoration.

## Verification

- [Final
CI](https://github.com/paperclipai/paperclip/actions/runs/39500401411)
passes on `89a8b5c290de3cfb9bdc061be3c88a29fd160a62`. Typecheck, build,
runner checks, server suites, and all browser shards pass.
- All four report browser regressions pass. They cover filters, gallery
navigation, repeated suite jumps, the folding sidebar, breadcrumbs,
scroll restoration, configuration selection and reload, mobile
navigation, and aggregate case status.
- The 63 report, history, preview-boundary, and navigation tests pass.
Runner E2E typecheck passes with `node node_modules/typescript/bin/tsc
-p tests/runner-e2e/tsconfig.json`.
- Manual browser checks covered desktop and a 390-pixel viewport.
Retained campaigns covered 209 executions in eight suites, 179
executions in seven suites, and 15 executions in one suite. Browser
fixture checks also confirmed filtered inspection and aggregate status.
No paid evaluations ran.
- The initial report browser tests still targeted the old layout and
routes. They were updated for the new navigation. Review also exposed
filtered-inspection and aggregate-status defects; both were fixed before
the final passing browser run.
- The unchanged Cases UI grouping test [failed once on the final
head](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118898559197).
Its full file also failed locally, while the isolated test passed. The
same UI code passed in earlier CI. A [single CI
retry](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118903333430)
passed all 3,978 UI tests and 286 CLI tests without code changes. The
cause remains unresolved.
- Two jobs on an intermediate head were interrupted by runner shutdown
signals. Those remain interrupted results. The complete final-head CI
run supersedes them for merge verification.
- Workspace `pnpm -r typecheck` and `pnpm build` also passed locally
before the rebase. The full local `pnpm test:run` attempt encountered
repeated embedded PostgreSQL startup failures and was stopped. It is not
a passing local full-suite result; final CI provides the full
verification.

## Risks

- Navigation and hash handling can select the wrong case or hide
results. Focused tests and browser checks cover these paths.
- Historical campaign links must resolve from both the preview and
published report locations.
- The report layout changes. Graders, prompts, retained outcomes, and
billing calculations do not change.

## Model Used

- OpenAI Codex with `gpt-6-astra` and `gpt-6.1-sol`. Used reasoning,
repository tools, code execution, and browser verification. The runtime
did not expose a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-20 16:50:53 -05:00

DottaandPaperclip

2b16262280

Improve runner E2E report navigation and local previews (#16996 )

## Thinking Path

> - Paperclip is the open source app that people use to manage AI agents
for work.
> - Product E2E campaigns check user journeys across agent profiles and
environments.
> - Readers use the campaign history and reports to inspect results and
retained evidence.
> - The reports showed image summaries, many unselected suites, and a
long sticky index.
> - Individual case cards also showed routine diagnostics before the
visual evidence.
> - This pull request adds text summaries, feature filters, folding case
indexes, and simpler cards.
> - A local preview uses retained public data so designers can change
reports without another paid campaign.

## Linked Issues or Issue Description

**Current behavior**

The report is hard to scan as the journey catalog grows. Suite links can
point into hidden results. Case indexes do not show whether a case
passed across its configurations. Design changes also need a realistic
local preview.

**Proposed behavior**

Show the selected campaign as text. Link journeys to current feature-map
capabilities. Add folding case indexes with status indicators, reliable
section links, and a back-to-top control. Keep failed outcomes visible
and put routine diagnostics under Test details. Render retained public
campaign data with local templates.

**Reason and benefit**

Readers can find a feature or failed case faster. Designers can iterate
against real retained data without starting an application instance or
using provider credentials.

**Breaking changes**

No result schema or grading change. The report stops embedding the
generated summary PNG. The publisher still produces that artifact for
existing consumers.

Related work: #12889 added the existing matrix browser. #12846 concerns
the public image publication boundary. This PR changes report
presentation and local preview behavior.

## What Changed

- Replace image summaries with shared HTML suite summaries on the
history and campaign pages.
- Show selected executions and preserve missing results for selected
cells.
- Add explicit journey-to-capability associations, feature filters, and
a coverage view with unmapped journeys.
- Add folding case indexes, section case lists, and red, green, or amber
status indicators across matching configurations.
- Fix anchor navigation through filters and add a floating back-to-top
control.
- Simplify case cards. Keep failure reasons visible. Fold routine
checks, billing, and context into Test details.
- Add an original-image link to the gallery.
- Add a local preview utility with public data caching and offline mode.
Document the direct Node command.
- Test manifest-listed evidence, rejection of unlisted files and encoded
traversal, cache reuse, and offline misses with a real loopback server
and a stub downloader.

## Verification

- Focused report, history, and preview tests: 75 passed in 6 files.
- Report Playwright spec: 4 passed in Chrome with a standalone config
that does not start the application server.
- `node scripts/check-maps.mjs`: passed.
- `git diff --check`: passed.
- Manually checked retained campaign reports, feature filters, folding
indexes, failure links, card details, gallery links, mobile layout, and
keyboard navigation.
- `pnpm build`: passed.
- `pnpm -r typecheck`: passed.
- `pnpm test:e2e:runner:typecheck`: passed.
- `pnpm test:run`: stopped after 15 native session database cases
failed. The local Postgres package first lacked dynamic-library links.
After link hydration, the same cases still failed. A direct `initdb`
check then reported `could not create semaphores: No space left on
device` and identified the macOS semaphore limit. The complete local
suite has not passed.
- `pnpm test:e2e:runner:unit`: 3,250 passed, 6 failed, 2 skipped. All
six failures reproduced against unchanged `origin/master` report sources
with the same installed dependencies. They concern Copilot admission and
its Daytona image input, plus the launcher test that runs those
assertions. No report or history test failed.
- No live provider evaluation was run for this presentation change.
- CI on `b1a9c06823` first failed browser shard 3/8. The agent-chat
split-handoff test reached cleanup, where cancelling a run returned HTTP
500 from a Postgres deadlock. Cleanup did not restore the experiment
setting, and a later test failed its initial setting check. This shard
does not contain the report spec. One targeted rerun passed, and that
full workflow then passed. This shows recovery; no production fix was
made for the deadlock. The later preview-test commit requires its own
fresh checks.

To review the design, run `node --watch --import
./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/preview.ts`.
Open the history address printed by the command, then open a campaign.
The README covers offline mode and cache refresh.

## Risks

- Feature associations use the current map. They describe linked
scenarios, not proof that a whole capability is covered. Historical
grades remain unchanged.
- New or unfamiliar cases remain unmapped until their assertions are
inspected.
- The preview needs the public history host on first access. Offline
mode requires cached files. It binds to the loopback interface and
serves manifest-listed public evidence.
- Hiding routine diagnostics adds one disclosure step. Failed or
incomplete outcomes remain visible.

This is report polish. It does not duplicate planned core work in
`ROADMAP.md`.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, code execution,
and browser automation. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-20 14:02:19 -05:00

DottaandPaperclip

4a7172f5ac

feat(runner-e2e): improve matrix report browsing (#12889 )

## Thinking Path

> - Paperclip is the open source app that people use to manage AI agents
for work.
> - The runner E2E system verifies agent profiles across supported
execution environments.
> - Its generated report is the main surface for inspecting those
results and their visual evidence.
> - Expanded matcher content could change the matrix column widths and
make comparisons difficult.
> - Screenshots also required extra navigation, and the report did not
have current-result search and filters.
> - This pull request stabilizes the matrix layout and makes visual
evidence directly browsable.
> - The benefit is faster inspection of retained test evidence without
another paid matrix run.

## Linked Issues or Issue Description

**What happened?**

The runner E2E report changed matrix column widths when a matcher table
expanded. The report also made screenshot comparison and current-result
discovery slower than necessary.

**Expected behavior**

The matrix columns must remain stable. Each retained screenshot must
appear as a thumbnail. The gallery must support keyboard navigation and
show the relevant execution metadata. The report must support
client-side search and filters.

**Steps to reproduce**

1. Open a runner E2E matrix report that contains retained screenshots.
2. Expand the matcher details in a matrix cell.
3. Observe the matrix column movement in the old report.

**Paperclip version or commit**

`8430bd897`

**Deployment mode**

Generated static runner E2E report.

## What Changed

- Keep matrix and matcher table widths stable when details expand.
- Show retained screenshot thumbnails in each test card.
- Add a full-screen evidence gallery with mouse, keyboard, and swipe
navigation.
- Show agent, environment, runtime, status, duration, token, and matcher
data in the gallery header.
- Add client-side search and profile, environment, suite, and status
filters below the report section tabs.
- Keep the filters in normal document flow while the report tabs remain
sticky.
- Add report generator assertions for the new layout and controls.
- Add Playwright coverage for filtering, stable matcher expansion, and
filtered gallery navigation.

## Verification

- `pnpm test:e2e:runner:typecheck`
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/report.test.ts`
- `PAPERCLIP_PLAYWRIGHT_CHANNEL=chrome pnpm exec playwright test
--config tests/e2e/playwright.config.ts
tests/e2e/runner-e2e-dashboard.spec.ts`
- `node --test ./scripts/__tests__/e2e-shard.test.mjs`
- `pnpm check:token-gates`
- `pnpm -r typecheck`
- `pnpm test:run`
- `pnpm build`
- Regenerated the report from GitHub Actions run `33963318820` without
rerunning the matrix.
- Verified 121 retained thumbnails across 66 results in a local browser.
- Verified that expanded matchers keep matrix widths at 260, 486, and
486 pixels in a 1280-pixel viewport.
- Verified search, filters, modal metadata, and arrow-key gallery
navigation.

## Risks

Low risk. This change only modifies the static runner E2E report
generator and its tests. It does not change runner execution or retained
evidence data.

> This work is focused report polish. It does not duplicate planned core
work in `ROADMAP.md`.

## Model Used

OpenAI Codex with `gpt-5.6-sol`. Codex desktop managed the context
window. The model used reasoning, filesystem tools, code execution,
GitHub CLI access, and in-app browser verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-09-05 07:46:05 -05:00