PaperClipAI

5,855 Commits 1,320 Branches 2,151 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	d8bd8fe9fb	feat(evals): browse E2E suites across targets and models (#18027 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - E2E tests show whether people can complete real app and agent journeys. > - Operators need to choose a suite and inspect its evidence across deployments and models. > - The history page listed campaigns without a suite browser or clear coverage filters. > - QA smoke also appeared separate from the other retained E2E measurements. > - This pull request puts suites, deployment targets, and named selections in one history browser. > - The benefit is that a passing subset stays distinct from a full suite run, with direct links to evidence. ## Linked Issues or Issue Description Related: #18009 added clear per-test details and links to the tested application. This PR extends the shared E2E history browser. What happened? The public directory called full app testing “Product E2E Evals” and made users choose campaigns before they could find a suite. The history list could not filter independently by tested app, agent execution, model, or run selection. A recent passing subset could be mistaken for a full suite result. Expected behavior Browse current and retained suites in one E2E category. Keep QA smoke in normal campaign history. Show the latest matching run and latest full suite run. Preserve the recorded definition and denominator when filters change. Steps to reproduce 1. Open the E2E history directory. 2. Find a suite, then compare a narrow passing run with a full run across execution environments and models. 3. Find the QA smoke measurement and open its retained screenshots. Paperclip version or commit The previous report UI was at `8279c99008`. This branch is rebased on current master. Deployment mode Static E2E history and campaign reports generated from retained measurements. ## What Changed - Call full-product testing “E2E tests.” Keep Runner protocol evals separate because they use a mock control plane. - Add a searchable suite sidebar that includes current suites and suites found only in retained history. - Add correlated suite, coverage, deployment target, execution, provider, model, profile, runtime, result, date, and named-selection filters. Keep filters in the URL for reload and back navigation. Sidebar suite links clear other filters; the Suite dropdown keeps deliberate intersections. Incompatible suite/preset URLs explain the empty selection and link to the suite runs. - Show latest matching and latest full run links. Distinguish full, subset, unfinished, and unknown coverage from the recorded suite definition and execution counts. Keep failed full runs visible. - Retain the tested application URL and runtime mode in new history records. Older records show “Not recorded.” Browser-only journeys do not gain an inferred agent or model. - Add an index-only refresh command that can import a reviewed public campaign once. It rejects private Cloud bundles and does not replace retained campaigns or upload evidence. - Keep existing report URLs, test IDs, commands, and measurement timestamps stable. Campaign links carry suite/profile/environment; target/model/provider/runtime remain history-level filters. - Add unit and synthetic browser regressions, including mobile navigation, long source refs, sidebar filter reset, and empty-selection repair. Inset dropdown carets in the shared report styles. Document direct Node entry points without adding package.json shortcuts. - Add a prominent Reset all filters button, blue active-filter borders and tint, and visible active counts for the main and advanced filters. Reset clears all 13 history filters, sidebar search, URL state, and the attention view. Campaign-page filters use the same active styling. ## Verification - Post-rebase focused Vitest: 127 tests passed across 10 files. The latest follow-up selection passed 100 tests across 9 files. E2E support TypeScript passed after both changes. - The synthetic Chromium journey covers full and subset coverage, unfinished selections, correlated filters, browser-only QA selection, URL/back/reload, and desktop and touch navigation. The expanded journey verifies 13 active history fields, the collapsed advanced-filter count, reset of every field and sidebar search, and active border color changes that revert after reset. It makes zero provider calls. - Hub Python tests: 7 passed. The browser harness also exits promptly when Chromium is unavailable; its HTTP server closes on launch failure. - Real history verification covered 419 campaigns and 59 suites. All 418 previous campaign records remained deeply equal after importing the reviewed QA campaign. Publication used the authoritative history ETag. Only mutable index/history/pointer files were written; immutable campaign bundles stayed unchanged. - A live browser walkthrough passed from the two-family eval hub through QA smoke, Basic app journeys, target filtering, the retained report, screenshot gallery, history, and the logo link. The QA URL was clickable and the image loaded. No browser JavaScript errors occurred. The original QA normalized results and all three PNG hashes remained unchanged. - Live [E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history), [core compatibility suite](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history&suite=core-compatibility), and [QA smoke runs](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history&preset=qa-smoke). - The live follow-up verified the incompatible Stock harness/QA-smoke URL, the repair link to 21 Stock harness campaigns, and sidebar navigation that clears preset/model filters. Dropdown carets were inspected in dark, light, and mobile views. No JavaScript errors occurred. - The final live follow-up verified reset of history controls, URL state, sidebar search, and the attention view, plus reload/back restoration and mobile touch reset. Active filter borders and tint were inspected in dark and light views. Campaign-page reset restored all three QA cases. All 419 history records and original QA measurement/evidence hashes stayed unchanged. - Full repository typecheck, test, build, and E2E CI gates passed on final head `1334cb79a5`. Both enforced aggregates, `ci / verify` and `ci / e2e`, completed successfully. Fresh Greptile review scored that exact head 5/5 with no unresolved review threads. ## Risks - Historical metadata can be missing. Unknown targets and runtime details stay explicit instead of being inferred from a worker environment. - Full suite coverage follows the saved definition and selection. Filtering to one model does not redefine that denominator. The latest full-run shortcut can cross coverage selections and says so in its detail text. - This changes shared report presentation. It adds no application runtime or database changes. The browser harness and index refresh do not invoke models. - Public history publication must keep its existing conditional-write and retained-evidence boundaries. Private bundles, raw logs, credentials, and screenshots are not part of this PR. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, code execution, and browser tools. The exact backend model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 09:48:31 -05:00
DottaandPaperclip	8279c99008	fix(evals): clarify report details and link tested applications (#18009 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E reports help operators inspect real application journeys. > - A worker environment does not identify the application that a journey tested. > - Reports hid that URL and gave missing diagnostics too much space. > - Standalone reports also sent the logo link to a missing parent page. > - This pull request shows the recorded target and checks in the shared report. > - The benefit is a clear report with useful links and inspectable diagnostics. ## Linked Issues or Issue Description What happened? A standalone Product E2E report had no application link in test details or the screenshot gallery. Its details showed several empty diagnostic sections before useful information. The logo could resolve to the page host root and return 404. Campaign notes added outside the report shell could overlap the fixed sidebar. Expected behavior Show the recorded application URL beside each test and screenshot. Put recorded checks before optional diagnostics. Keep campaign context inside the shared layout. Link the logo to the evaluations directory. Steps to reproduce 1. Render a retained campaign at a standalone page path without a history index. 2. Open a test with screenshot evidence and missing startup timing. 3. Inspect test details and gallery information, then click the report logo. Paperclip version or commit Reproduced in the report renderer at `39e811c9c5`. This branch applies the fix to current master. Deployment mode Static Product E2E HTML report generated from retained normalized results. ## What Changed - Add optional recorded `targetUrl` metadata and resolve existing Product E2E and provider-connection targets. Only HTTP and HTTPS links without embedded credentials are shown. - Show the target URL in test details and screenshot gallery information. - Put recorded checks and execution facts first. Keep raw usage, timing, and startup details under Diagnostics. Preserve missing metadata as unknown. - Keep campaign titles, descriptions, notes, and evidence links inside the shared report layout during retained-evidence regeneration. - Fix standalone navigation and let the brand lockup wrap at narrow sidebar widths. - Match report profiles by suite and recorded provider/model before falling back to an inferred profile. - Add report regressions and document the retained metadata contract. ## Verification - Focused report tests: 86 passed across 8 files on the isolated branch. Command: `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts report-target.test.ts report-navigation.test.ts report-catalog.test.ts first-task-report.test.ts report.test.ts dashboard-regenerate.test.ts`. - Browser verification covered dark desktop widths 1440 and 1000, light intermediate width 780, and dark mobile width 390. It covered the drawer, URL links, gallery keyboard navigation, focus return, and the logo destination. - The published retained report keeps all 3 original passing cases and 7 recorded checks. Its campaign and screenshot bytes remain unchanged. This presentation refresh made zero provider calls. - Full `pnpm build`, `pnpm -r typecheck`, and the eval TypeScript check passed after rebuilding the reused package dependencies. The first typecheck attempt had stale local dependency builds; those errors are resolved. - The local `pnpm test:run` attempt hit repeated embedded PostgreSQL bootstrap failures in server suites. I stopped that attempt and retained its log. It is not counted as a passing check. Current-head CI remains the full test gate. - Review follow-ups add provider/model and cross-suite profile regressions. Malformed campaign presentation metadata is rejected before any report or retained-result writes; ten malformed input cases verify that retained results stay unchanged. ## Risks - This changes presentation across Product E2E reports. Historical reports without a target show “Not recorded” rather than an inferred URL. - Unknown startup reuse data remains in raw metadata but no longer creates an empty table. - Standalone logo links open the shared public evaluations directory. Campaign history links still use a matching history index when present. - No application runtime, database, or measurement verdict changes are included. Private QA configuration and raw campaign evidence stay outside this PR. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, code execution, and browser tools. The exact backend model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 13:00:54 +00:00
DottaandPaperclip	f89c37791c	Add campaign sidebar and case navigation to runner E2E reports (#17010 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E campaigns test runner behavior across agent profiles and environments. > - The retained reports contain many suites, cases, and execution previews. > - A long report makes it hard to find a case or return to the campaign context. > - This pull request adds campaign navigation, case breadcrumbs, and separate global history pages. > - Users can scan all execution previews or inspect one case without losing their place. ## Linked Issues or Issue Description What existing behavior does this improve? Navigation in the retained Runner E2E campaign reports and history homepage. Subsystem affected Test report infrastructure in `tests/runner-e2e/`. Current behavior Reports show a long configuration matrix. The suite index and campaign history share the same page. Case links do not provide a focused inspection view. Proposed behavior Global overview, history, and attention pages show global navigation. A specific campaign adds a folding suite tree. All results keeps the full scrollable report. A case opens with breadcrumbs and an execution picker. Reason and benefit Users can find a case, compare its executions, and return to their reading position. Global pages do not show suites from an unrelated campaign. Breaking changes The report HTML layout changes. Retained result schemas, execution identities, and suite and execution anchors remain unchanged. Additional context Refs: #16996. This extends the merged report navigation and local preview work. The duplicate PR and issue search found no open PR for this change. ## What Changed - Add shared report navigation with the Paperclip brand mark and wordmark. - Scope the suite tree to the current campaign. Keep global overview, history, and attention pages separate. - Group all execution previews by suite and case. Reuse the existing evidence cards and image gallery. - Add focused case inspection, configuration selection, breadcrumbs, filter handling, and scroll restoration. - Move historical trends to the global history page. Keep billing details, missing results, and failure evidence available. - Update report documentation and browser regressions for filters, status dots, configuration reloads, breadcrumbs, and scroll restoration. ## Verification - [Final CI](https://github.com/paperclipai/paperclip/actions/runs/39500401411) passes on `89a8b5c290de3cfb9bdc061be3c88a29fd160a62`. Typecheck, build, runner checks, server suites, and all browser shards pass. - All four report browser regressions pass. They cover filters, gallery navigation, repeated suite jumps, the folding sidebar, breadcrumbs, scroll restoration, configuration selection and reload, mobile navigation, and aggregate case status. - The 63 report, history, preview-boundary, and navigation tests pass. Runner E2E typecheck passes with `node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json`. - Manual browser checks covered desktop and a 390-pixel viewport. Retained campaigns covered 209 executions in eight suites, 179 executions in seven suites, and 15 executions in one suite. Browser fixture checks also confirmed filtered inspection and aggregate status. No paid evaluations ran. - The initial report browser tests still targeted the old layout and routes. They were updated for the new navigation. Review also exposed filtered-inspection and aggregate-status defects; both were fixed before the final passing browser run. - The unchanged Cases UI grouping test [failed once on the final head](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118898559197). Its full file also failed locally, while the isolated test passed. The same UI code passed in earlier CI. A [single CI retry](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118903333430) passed all 3,978 UI tests and 286 CLI tests without code changes. The cause remains unresolved. - Two jobs on an intermediate head were interrupted by runner shutdown signals. Those remain interrupted results. The complete final-head CI run supersedes them for merge verification. - Workspace `pnpm -r typecheck` and `pnpm build` also passed locally before the rebase. The full local `pnpm test:run` attempt encountered repeated embedded PostgreSQL startup failures and was stopped. It is not a passing local full-suite result; final CI provides the full verification. ## Risks - Navigation and hash handling can select the wrong case or hide results. Focused tests and browser checks cover these paths. - Historical campaign links must resolve from both the preview and published report locations. - The report layout changes. Graders, prompts, retained outcomes, and billing calculations do not change. ## Model Used - OpenAI Codex with `gpt-6-astra` and `gpt-6.1-sol`. Used reasoning, repository tools, code execution, and browser verification. The runtime did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 16:50:53 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

d8bd8fe9fb

feat(evals): browse E2E suites across targets and models (#18027 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - E2E tests show whether people can complete real app and agent
journeys.
> - Operators need to choose a suite and inspect its evidence across
deployments and models.
> - The history page listed campaigns without a suite browser or clear
coverage filters.
> - QA smoke also appeared separate from the other retained E2E
measurements.
> - This pull request puts suites, deployment targets, and named
selections in one history browser.
> - The benefit is that a passing subset stays distinct from a full
suite run, with direct links to evidence.

## Linked Issues or Issue Description

Related: #18009 added clear per-test details and links to the tested
application. This PR extends the shared E2E history browser.

**What happened?**

The public directory called full app testing “Product E2E Evals” and
made users choose campaigns before they could find a suite. The history
list could not filter independently by tested app, agent execution,
model, or run selection. A recent passing subset could be mistaken for a
full suite result.

**Expected behavior**

Browse current and retained suites in one E2E category. Keep QA smoke in
normal campaign history. Show the latest matching run and latest full
suite run. Preserve the recorded definition and denominator when filters
change.

**Steps to reproduce**

1. Open the E2E history directory.
2. Find a suite, then compare a narrow passing run with a full run
across execution environments and models.
3. Find the QA smoke measurement and open its retained screenshots.

**Paperclip version or commit**

The previous report UI was at `8279c99008`. This branch is rebased on
current master.

**Deployment mode**

Static E2E history and campaign reports generated from retained
measurements.

## What Changed

- Call full-product testing “E2E tests.” Keep Runner protocol evals
separate because they use a mock control plane.
- Add a searchable suite sidebar that includes current suites and suites
found only in retained history.
- Add correlated suite, coverage, deployment target, execution,
provider, model, profile, runtime, result, date, and named-selection
filters. Keep filters in the URL for reload and back navigation. Sidebar
suite links clear other filters; the Suite dropdown keeps deliberate
intersections. Incompatible suite/preset URLs explain the empty
selection and link to the suite runs.
- Show latest matching and latest full run links. Distinguish full,
subset, unfinished, and unknown coverage from the recorded suite
definition and execution counts. Keep failed full runs visible.
- Retain the tested application URL and runtime mode in new history
records. Older records show “Not recorded.” Browser-only journeys do not
gain an inferred agent or model.
- Add an index-only refresh command that can import a reviewed public
campaign once. It rejects private Cloud bundles and does not replace
retained campaigns or upload evidence.
- Keep existing report URLs, test IDs, commands, and measurement
timestamps stable. Campaign links carry suite/profile/environment;
target/model/provider/runtime remain history-level filters.
- Add unit and synthetic browser regressions, including mobile
navigation, long source refs, sidebar filter reset, and empty-selection
repair. Inset dropdown carets in the shared report styles. Document
direct Node entry points without adding package.json shortcuts.
- Add a prominent Reset all filters button, blue active-filter borders
and tint, and visible active counts for the main and advanced filters.
Reset clears all 13 history filters, sidebar search, URL state, and the
attention view. Campaign-page filters use the same active styling.

## Verification

- Post-rebase focused Vitest: 127 tests passed across 10 files. The
latest follow-up selection passed 100 tests across 9 files. E2E support
TypeScript passed after both changes.
- The synthetic Chromium journey covers full and subset coverage,
unfinished selections, correlated filters, browser-only QA selection,
URL/back/reload, and desktop and touch navigation. The expanded journey
verifies 13 active history fields, the collapsed advanced-filter count,
reset of every field and sidebar search, and active border color changes
that revert after reset. It makes zero provider calls.
- Hub Python tests: 7 passed. The browser harness also exits promptly
when Chromium is unavailable; its HTTP server closes on launch failure.
- Real history verification covered 419 campaigns and 59 suites. All 418
previous campaign records remained deeply equal after importing the
reviewed QA campaign. Publication used the authoritative history ETag.
Only mutable index/history/pointer files were written; immutable
campaign bundles stayed unchanged.
- A live browser walkthrough passed from the two-family eval hub through
QA smoke, Basic app journeys, target filtering, the retained report,
screenshot gallery, history, and the logo link. The QA URL was clickable
and the image loaded. No browser JavaScript errors occurred. The
original QA normalized results and all three PNG hashes remained
unchanged.
- Live [E2E
history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history),
[core compatibility
suite](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history&suite=core-compatibility),
and [QA smoke
runs](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/?view=history&preset=qa-smoke).
- The live follow-up verified the incompatible Stock harness/QA-smoke
URL, the repair link to 21 Stock harness campaigns, and sidebar
navigation that clears preset/model filters. Dropdown carets were
inspected in dark, light, and mobile views. No JavaScript errors
occurred.
- The final live follow-up verified reset of history controls, URL
state, sidebar search, and the attention view, plus reload/back
restoration and mobile touch reset. Active filter borders and tint were
inspected in dark and light views. Campaign-page reset restored all
three QA cases. All 419 history records and original QA
measurement/evidence hashes stayed unchanged.
- Full repository typecheck, test, build, and E2E CI gates passed on
final head `1334cb79a5`. Both enforced aggregates, `ci / verify` and `ci
/ e2e`, completed successfully. Fresh Greptile review scored that exact
head 5/5 with no unresolved review threads.

## Risks

- Historical metadata can be missing. Unknown targets and runtime
details stay explicit instead of being inferred from a worker
environment.
- Full suite coverage follows the saved definition and selection.
Filtering to one model does not redefine that denominator. The latest
full-run shortcut can cross coverage selections and says so in its
detail text.
- This changes shared report presentation. It adds no application
runtime or database changes. The browser harness and index refresh do
not invoke models.
- Public history publication must keep its existing conditional-write
and retained-evidence boundaries. Private bundles, raw logs,
credentials, and screenshots are not part of this PR.

## Model Used

- OpenAI Codex, GPT-6 family, with reasoning, code execution, and
browser tools. The exact backend model ID and context-window size are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-28 09:48:31 -05:00

DottaandPaperclip

8279c99008

fix(evals): clarify report details and link tested applications (#18009 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E reports help operators inspect real application
journeys.
> - A worker environment does not identify the application that a
journey tested.
> - Reports hid that URL and gave missing diagnostics too much space.
> - Standalone reports also sent the logo link to a missing parent page.
> - This pull request shows the recorded target and checks in the shared
report.
> - The benefit is a clear report with useful links and inspectable
diagnostics.

## Linked Issues or Issue Description

**What happened?**

A standalone Product E2E report had no application link in test details
or the screenshot gallery. Its details showed several empty diagnostic
sections before useful information. The logo could resolve to the page
host root and return 404. Campaign notes added outside the report shell
could overlap the fixed sidebar.

**Expected behavior**

Show the recorded application URL beside each test and screenshot. Put
recorded checks before optional diagnostics. Keep campaign context
inside the shared layout. Link the logo to the evaluations directory.

**Steps to reproduce**

1. Render a retained campaign at a standalone page path without a
history index.
2. Open a test with screenshot evidence and missing startup timing.
3. Inspect test details and gallery information, then click the report
logo.

**Paperclip version or commit**

Reproduced in the report renderer at `39e811c9c5`. This branch applies
the fix to current master.

**Deployment mode**

Static Product E2E HTML report generated from retained normalized
results.

## What Changed

- Add optional recorded `targetUrl` metadata and resolve existing
Product E2E and provider-connection targets. Only HTTP and HTTPS links
without embedded credentials are shown.
- Show the target URL in test details and screenshot gallery
information.
- Put recorded checks and execution facts first. Keep raw usage, timing,
and startup details under Diagnostics. Preserve missing metadata as
unknown.
- Keep campaign titles, descriptions, notes, and evidence links inside
the shared report layout during retained-evidence regeneration.
- Fix standalone navigation and let the brand lockup wrap at narrow
sidebar widths.
- Match report profiles by suite and recorded provider/model before
falling back to an inferred profile.
- Add report regressions and document the retained metadata contract.

## Verification

- Focused report tests: 86 passed across 8 files on the isolated branch.
Command: `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts report-target.test.ts
report-navigation.test.ts report-catalog.test.ts
first-task-report.test.ts report.test.ts dashboard-regenerate.test.ts`.
- Browser verification covered dark desktop widths 1440 and 1000, light
intermediate width 780, and dark mobile width 390. It covered the
drawer, URL links, gallery keyboard navigation, focus return, and the
logo destination.
- The published retained report keeps all 3 original passing cases and 7
recorded checks. Its campaign and screenshot bytes remain unchanged.
This presentation refresh made zero provider calls.
- Full `pnpm build`, `pnpm -r typecheck`, and the eval TypeScript check
passed after rebuilding the reused package dependencies. The first
typecheck attempt had stale local dependency builds; those errors are
resolved.
- The local `pnpm test:run` attempt hit repeated embedded PostgreSQL
bootstrap failures in server suites. I stopped that attempt and retained
its log. It is not counted as a passing check. Current-head CI remains
the full test gate.
- Review follow-ups add provider/model and cross-suite profile
regressions. Malformed campaign presentation metadata is rejected before
any report or retained-result writes; ten malformed input cases verify
that retained results stay unchanged.

## Risks

- This changes presentation across Product E2E reports. Historical
reports without a target show “Not recorded” rather than an inferred
URL.
- Unknown startup reuse data remains in raw metadata but no longer
creates an empty table.
- Standalone logo links open the shared public evaluations directory.
Campaign history links still use a matching history index when present.
- No application runtime, database, or measurement verdict changes are
included. Private QA configuration and raw campaign evidence stay
outside this PR.

## Model Used

- OpenAI Codex, GPT-6 family, with reasoning, code execution, and
browser tools. The exact backend model ID and context-window size are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-28 13:00:54 +00:00

DottaandPaperclip

f89c37791c

Add campaign sidebar and case navigation to runner E2E reports (#17010 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E campaigns test runner behavior across agent profiles and
environments.
> - The retained reports contain many suites, cases, and execution
previews.
> - A long report makes it hard to find a case or return to the campaign
context.
> - This pull request adds campaign navigation, case breadcrumbs, and
separate global history pages.
> - Users can scan all execution previews or inspect one case without
losing their place.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Navigation in the retained Runner E2E campaign reports and history
homepage.

**Subsystem affected**

Test report infrastructure in `tests/runner-e2e/`.

**Current behavior**

Reports show a long configuration matrix. The suite index and campaign
history share the same page. Case links do not provide a focused
inspection view.

**Proposed behavior**

Global overview, history, and attention pages show global navigation. A
specific campaign adds a folding suite tree. All results keeps the full
scrollable report. A case opens with breadcrumbs and an execution
picker.

**Reason and benefit**

Users can find a case, compare its executions, and return to their
reading position. Global pages do not show suites from an unrelated
campaign.

**Breaking changes**

The report HTML layout changes. Retained result schemas, execution
identities, and suite and execution anchors remain unchanged.

**Additional context**

Refs: #16996. This extends the merged report navigation and local
preview work. The duplicate PR and issue search found no open PR for
this change.

## What Changed

- Add shared report navigation with the Paperclip brand mark and
wordmark.
- Scope the suite tree to the current campaign. Keep global overview,
history, and attention pages separate.
- Group all execution previews by suite and case. Reuse the existing
evidence cards and image gallery.
- Add focused case inspection, configuration selection, breadcrumbs,
filter handling, and scroll restoration.
- Move historical trends to the global history page. Keep billing
details, missing results, and failure evidence available.
- Update report documentation and browser regressions for filters,
status dots, configuration reloads, breadcrumbs, and scroll restoration.

## Verification

- [Final
CI](https://github.com/paperclipai/paperclip/actions/runs/39500401411)
passes on `89a8b5c290de3cfb9bdc061be3c88a29fd160a62`. Typecheck, build,
runner checks, server suites, and all browser shards pass.
- All four report browser regressions pass. They cover filters, gallery
navigation, repeated suite jumps, the folding sidebar, breadcrumbs,
scroll restoration, configuration selection and reload, mobile
navigation, and aggregate case status.
- The 63 report, history, preview-boundary, and navigation tests pass.
Runner E2E typecheck passes with `node node_modules/typescript/bin/tsc
-p tests/runner-e2e/tsconfig.json`.
- Manual browser checks covered desktop and a 390-pixel viewport.
Retained campaigns covered 209 executions in eight suites, 179
executions in seven suites, and 15 executions in one suite. Browser
fixture checks also confirmed filtered inspection and aggregate status.
No paid evaluations ran.
- The initial report browser tests still targeted the old layout and
routes. They were updated for the new navigation. Review also exposed
filtered-inspection and aggregate-status defects; both were fixed before
the final passing browser run.
- The unchanged Cases UI grouping test [failed once on the final
head](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118898559197).
Its full file also failed locally, while the isolated test passed. The
same UI code passed in earlier CI. A [single CI
retry](https://github.com/paperclipai/paperclip/actions/runs/39500401411/job/118903333430)
passed all 3,978 UI tests and 286 CLI tests without code changes. The
cause remains unresolved.
- Two jobs on an intermediate head were interrupted by runner shutdown
signals. Those remain interrupted results. The complete final-head CI
run supersedes them for merge verification.
- Workspace `pnpm -r typecheck` and `pnpm build` also passed locally
before the rebase. The full local `pnpm test:run` attempt encountered
repeated embedded PostgreSQL startup failures and was stopped. It is not
a passing local full-suite result; final CI provides the full
verification.

## Risks

- Navigation and hash handling can select the wrong case or hide
results. Focused tests and browser checks cover these paths.
- Historical campaign links must resolve from both the preview and
published report locations.
- The report layout changes. Graders, prompts, retained outcomes, and
billing calculations do not change.

## Model Used

- OpenAI Codex with `gpt-6-astra` and `gpt-6.1-sol`. Used reasoning,
repository tools, code execution, and browser verification. The runtime
did not expose a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-20 16:50:53 -05:00