mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 11:20:42 +01:00
e6ec781a1a
## Thinking Path > - Paperclip's Product E2E journeys verify user outcomes in the real app and GitHub. > - The review journey must show the score and failed check for its exact reviewed commit, then the visible task status before and after reload. > - GitHub includes a hidden “Re-run failed checks” menu item before the visible check result. > - The journey selected that hidden item and failed even after native work and publication completed. A subsequent attempt exposed the same first-text-match problem with Done in the task UI. > - This change reads the selected check's status header and binds its URL, score, and commit. > - Browser fixtures retain this failure and reject plausible wrong projections. ## Linked Issues or Issue Description Refs: #17457, #17513, #17551. **What happened?** The bot completed its native run and task, then published its summary, inline precision finding, and failed 2/5 check. The browser assertion selected a hidden “Re-run failed checks” menu item before the visible result, so the journey still failed. After that was corrected, the next live attempt passed the check projection but selected hidden Done text before the visible task status. **Expected behavior** Verify the actual selected GitHub check. A hidden menu item or a word in the model's narrative cannot prove its conclusion. **Steps to reproduce** 1. Render a hidden “Re-run failed checks” item before a visible failed check. 2. Use the old first text match for “failed”. 3. Observe the false visibility failure. **Paperclip version or commit** The base includes the journey changes from #17457 and #17513. ## What Changed - Validate the check URL against the configured repository and select its exact check-run ID. - Require the expected score and full reviewed commit inside that check. - Read failure status only from that check's header. - Give each projection step its own phase so failures identify the correct boundary. - Stage the verified persisted screenshot as the standard report final-state artifact, rejecting a passing report if it is absent. - Preserve measured billing summaries during regeneration when raw run usage is absent; newer raw usage takes precedence. - Extend the catalog-wide history fixture to include the new journey evidence contract. - Read task completion from the visible Status property picker before and after reload, with separate phase labels. - Add eighteen isolated browser fixtures to the existing browser-support suite and document both boundaries. ## Verification - Reproduced the old selector failure in the live GitHub page and a credential-free browser fixture before fixing it. - All 60 browser-support cases pass, including eighteen focused projection cases. They reject the wrong check status, score, commit, page, and repository reference, plus hidden or unrelated Done text. - All 163 focused billing, history, accounting, report, and journey tests pass, as does runner E2E typechecking. - A regression reproduced discarded saved token totals. Retained totals now survive per-cell and campaign aggregation while missing dollar prices stay unknown; fresh raw usage still overrides stale summaries. - Diff and credential-value checks pass. - The earlier failed attempts remain failed. A fresh complete QA journey passed all 18 checkpoints and cleanup with a successful native run, a below-threshold review and inline precision finding, the correctly failed check, visible Done status, and reload persistence. The later reference-validation commit changes only browser fixtures; the live driver bytes are unchanged. - A report regression reproduced the missing standard final-state artifact. The packager now aliases only the hash-verified persisted screenshot for passing results; missing final-state proof is rejected. Presentation regeneration preserves the original live grade and dispatches no provider work. ## Risks This depends on GitHub's check-run header structure. A markup change fails the journey instead of accepting narrative text as a result. The product runtime and behavioral oracle stay unchanged. Usage-price coverage remains a separate evidence dimension. A saved billing summary is reused only when raw usage is absent, preserving the recorded coverage and unknown prices. The separate advisory Paperclip Review remains queued after its prior assigned attempt timed out. Its normal Retry control rejected the obsolete failed chat request as unsafe to retry. Required repository gates `ci / verify` and `ci / e2e` pass, all 55 completed CI checks are green, and final-head Greptile is 5/5 with no unresolved findings. The advisory queue is not represented as a completed assessment. ## Model Used OpenAI Codex, based on GPT-6. The exact runtime model ID and context-window size are not exposed in this session. Used for code, browser fixtures, and live evidence analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <paperclip@paperclip.ing> Co-authored-by: Paperclip <bot@paperclip.ing>
19 lines
852 B
TypeScript
19 lines
852 B
TypeScript
import { defineConfig } from "@playwright/test";
|
|
|
|
/** Browser-only harness regressions: no Paperclip instance or provider credentials. */
|
|
export default defineConfig({
|
|
testDir: ".",
|
|
testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "github-review-check.spec.ts"],
|
|
workers: 1,
|
|
retries: 0,
|
|
timeout: 10_000,
|
|
use: {
|
|
headless: true,
|
|
...(process.env.PAPERCLIP_PLAYWRIGHT_CHANNEL
|
|
? { channel: process.env.PAPERCLIP_PLAYWRIGHT_CHANNEL }
|
|
: {}),
|
|
},
|
|
outputDir: "./results/browser-support",
|
|
reporter: "list",
|
|
});
|