Files
PaperClipAI/tests/runner-e2e/github-review-browser.ts
T
DottaandPaperclip 6d46851920 test(e2e): add attended GitHub review bot journey (#17457)
## Thinking Path

> - Paperclip manages agents and the work they perform.
> - Product E2E checks verify real user paths through those agents.
> - A recorded GitHub review-bot walkthrough had no repeatable
registered case.
> - App setup, automatic triggering, publication, and task completion
need separate evidence.
> - This pull request adds an explicit attended journey to the existing
catalog and report tools.
> - Reusable setup profiles and independent assertions preserve the same
outcome across compatible targets.

## Linked Issues or Issue Description

Refs: #17262. This case remains an explicit attended attached-target
flow. It does not replace the shared deployment-target or
paid-operations framework.

**What existing behavior does this improve?**

The Product E2E catalog can now select and retain evidence for GitHub
review-bot setup and its first automatic review.

**Current behavior**

Recorded walkthroughs cannot be rerun as a fixed registered case. A
published review alone can hide a failed run or unfinished task.

**Proposed behavior**

Create and install an owned App, save automatic new-PR settings, create
a defective PR, and require correlated review delivery, an inline
finding, a consistent check, a successful run, and a Done task. Retain
failures and confirm owned-resource cleanup.

**Reason and benefit**

The case makes a useful external integration reproducible without
treating setup, unit checks, or report rendering as proof of a completed
live journey.

## What Changed

- Add two explicit attended cells for an existing configured agent in
local or Daytona execution.
- Declare hosting, account, connection, and credential prerequisites
through private setup bindings.
- Add fixed browser controls, ownership checks, independent grading, and
cleanup.
- Package declared screenshots through the existing sanitizer, manifest,
validator, and dashboard.
- Reject operations-managed execution until budgeted routing is
qualified.
- Document hosted QA credential setup and evidence limits.

## Verification

- Focused catalog, oracle, report, connection, and ownership checks: 127
tests passed.
- Inherited-environment preflight regression failed against the previous
commit and passed after the fix. Actual-run environment verification
remains required.
- Harness TypeScript check and full workspace typecheck passed. Full
build passed. The local full-test attempt was interrupted after repeated
embedded PostgreSQL startup failures; its exit code of zero does not
establish a pass; Exact-head Linux CI passed the full test, typecheck,
build, Runner, serialized, browser, and canary gates on
`abbddeade848edefc8de75f11cc9008b2cd52764`.
- A real hosted attempt published a rated review and inline finding. It
failed when its native session timed out before the task reached Done.
Cleanup passed. Full live qualification remains open.
- Credential scan found no known credential values in the authored files
or retained report corpus.

## Risks

- The full case creates external test resources and uses the configured
provider. It requires an explicit selection and current task authority.
It is excluded from default and CI matrices.
- Time and run limits are not a hard dollar cap. Missing dollar cost
stays unknown. Fixture setup does not weaken budget policy.
- Fresh personal OAuth, unattended authentication, runtime skill-read
proof, and cross-host qualification are separate claims.
- The selected live fixture needs environment/lifecycle recovery before
it can adopt a longer provider deadline.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, code execution, and
browser tools. The exact deployment model ID and context-window size are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, including the available version
and capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the enhancement
template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open findings
- [x] I will address all Greptile and reviewer comments before merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 12:16:26 -07:00

182 lines
12 KiB
TypeScript

import { expect, type Page } from "@playwright/test";
import { ConnectionBlock, ConnectionFailure } from "./connection-config.js";
import { connectionDelay } from "./connection-target.js";
import type { GitHubReviewJourneyConfig } from "./github-review-config.js";
export async function enterGitHubJourneyCompany(page: Page, origin: string, company: { issuePrefix: string }) {
if (!/^[A-Za-z0-9]+$/.test(company.issuePrefix)) throw new ConnectionFailure("invalid_journey_company_prefix");
const pathname = `/${company.issuePrefix}/dashboard`;
await page.goto(`${origin}${pathname}`);
await page.waitForURL(url => url.origin === origin && url.pathname === pathname);
}
/** Capture measured pages only, excluding authentication and unrelated GitHub data. */
export function isGitHubJourneyEvidencePage(value: string, targetOrigin: string, repository: string, appSlug?: string) {
const url = new URL(value);
if ([...url.searchParams.keys()].some(key => /token|secret|code|state|password/i.test(key))) return false;
if (url.origin === targetOrigin) return !/connect|login|auth/.test(url.pathname);
if (url.origin !== "https://github.com") return false;
const prefix = `/${repository}/`;
if (url.pathname.startsWith(prefix) && /^(?:pull\/\d+(?:\/(?:files|commits|checks))?|runs\/\d+|actions\/runs\/\d+(?:\/job\/\d+)?)\/?$/.test(url.pathname.slice(prefix.length))) return true;
return Boolean(appSlug && (url.pathname === `/apps/${appSlug}/installations/new` || url.pathname === `/apps/${appSlug}/installations/new/permissions`));
}
export async function awaitJourneyCondition(check: () => Promise<boolean>, timeoutMs: number, code: string, signal: AbortSignal, boundary: "awaiting_user" | "failed" = "awaiting_user") {
const end = Date.now() + timeoutMs;
while (Date.now() < end) {
if (signal.aborted) throw new ConnectionBlock("blocked_target", "journey_interrupted");
if (await check()) return;
await connectionDelay(750);
}
if (boundary === "failed") throw new ConnectionFailure(code);
throw new ConnectionBlock("awaiting_user", code);
}
export async function observeGitHubJourneyAccount(page: Page) {
const menu = page.getByRole("button", { name: /Open user navigation menu|View profile and more/i });
const dialog = page.getByRole("dialog", { name: "User navigation", exact: true });
await menu.click();
try { await dialog.waitFor({ timeout: 3_000 }); }
catch (error) {
// GitHub can replace its server-rendered header during hydration. Retry
// once only when the observed control still explicitly says Open.
if (!await menu.isVisible() || await dialog.isVisible()) throw error;
await menu.click();
await dialog.waitFor();
}
const profile = dialog.getByRole("link", { name: "Profile", exact: true });
const href = await profile.getAttribute("href");
const url = new URL(href ?? "", "https://github.com");
const account = url.origin === "https://github.com" ? /^\/([A-Za-z0-9-]+)\/?$/.exec(url.pathname)?.[1] : undefined;
if (!account) throw new ConnectionFailure("github_profile_identity_missing");
await page.keyboard.press("Escape");
return account;
}
/** Prepare one repository grant; submitting Install remains a separate action. */
export async function selectGitHubJourneyRepository(page: Page, repository: string) {
const selectedOnly = page.getByRole("radio", { name: /Only select repositories/i });
await selectedOnly.check();
await page.getByRole("button", { name: /Select repositories/i }).click();
await page.getByRole("textbox", { name: "Search for a repository", exact: true }).fill(repository.split("/")[1]!);
const name = new RegExp(`^${repository.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}(?:\\s|$)`);
// GitHub nests the selectable repository item inside a containing menu item.
await page.getByRole("menuitem", { name }).last().click();
await page.keyboard.press("Escape");
await expect(selectedOnly).toBeChecked();
await expect(page.getByRole("radio", { name: /^All repositories/ })).not.toBeChecked();
await expect(page.getByRole("main")).toContainText("Selected 1 repository.");
await expect(page.getByRole("button", { name: `Remove ${repository.split("/")[1]}`, exact: true })).toBeVisible();
await expect(page.getByRole("button", { name: "Install", exact: true })).toBeEnabled();
}
/** Login remains ordinary browser UI. Passkeys are never exported or emulated. */
export async function ensureGitHubJourneyLogin(input: {
page: Page; config: GitHubReviewJourneyConfig; username: string; password?: () => Promise<string>; signal: AbortSignal;
checkpoint(id: string): Promise<void>;
}) {
const { page, config } = input;
const signedIn = () => page.getByRole("button", { name: /Open user navigation menu|View profile and more/i }).isVisible();
if (!await signedIn()) {
const loginLink = page.getByRole("link", { name: /^Sign in$/ });
if (await loginLink.isVisible()) await loginLink.click();
if (config.github.login === "password") {
await page.getByLabel("Username or email address", { exact: true }).fill(input.username);
await page.getByLabel("Password", { exact: true }).fill(await input.password!());
await page.getByRole("button", { name: "Sign in", exact: true }).click();
}
await input.checkpoint("awaiting_github_sign_in");
console.log("[github-review-journey] Complete Ralph QA GitHub sign-in/passkey in the dedicated QA browser. No credential needs to be pasted into chat.");
await awaitJourneyCondition(signedIn, config.browser.loginTimeoutMs, "github_login_required", input.signal);
}
// The login identifier can be an email address; verify the GitHub handle.
if ((await observeGitHubJourneyAccount(page)).toLowerCase() !== config.github.account.toLowerCase()) throw new ConnectionBlock("blocked_target", "github_account_mismatch");
}
/** Explicit one-time bootstrap. Later attempts reuse this private repository. */
export async function createPrivateJourneyRepository(page: Page, repository: string) {
const [owner, name] = repository.split("/");
await page.getByRole("button", { name: /Create new/i }).click();
await page.getByRole("menuitem", { name: /New repository/i }).click();
const ownerButton = page.getByRole("button", { name: /Owner|Choose an owner/i });
await expect(ownerButton).toHaveText(owner!);
await page.getByRole("textbox", { name: /Repository name/i }).fill(name!);
await expect(page.getByText(`${name} is available.`, { exact: true })).toBeVisible();
const privateChoice = page.getByRole("radio", { name: /Private/ });
if (await privateChoice.isVisible()) await privateChoice.check();
else {
await page.getByRole("button", { name: /Choose visibility|Public/i }).click();
await page.getByRole("menuitemradio", { name: /Private/ }).click();
}
await expect(page.getByRole("button", { name: "Choose visibility: Private", exact: true })).toBeVisible();
const readme = page.getByRole("button", { name: "Add README", exact: true });
if (await readme.getAttribute("aria-pressed") !== "true") await readme.click();
await expect(readme).toHaveAttribute("aria-pressed", "true");
await page.getByRole("button", { name: "Create repository", exact: true }).click();
await page.waitForURL(url => url.origin === "https://github.com" && url.pathname === `/${repository}`);
await expect(page.getByRole("link", { name: name!, exact: true }).first()).toBeVisible();
await expect(page.getByText("Private", { exact: true }).first()).toBeVisible();
}
export async function createGitHubJourneyPullRequest(input: { page: Page; branch: string; title: string; contents: string }) {
const { page } = input;
// The initial file list is server-rendered before GitHub's controls hydrate.
// Its loaded commit receipt provides a visible readiness signal.
await expect(page.getByRole("link", { name: /^Commit [a-f0-9]{7,40}$/ })).toBeVisible();
await page.getByRole("button", { name: "Add file", exact: true }).click();
await page.getByRole("menuitem", { name: "Create new file", exact: true }).click();
await page.getByRole("textbox", { name: /Name your file|File name/i }).fill("cart-total.js");
await page.getByRole("textbox", { name: /Editing cart-total.js file contents|File contents/i }).fill(input.contents);
await page.getByRole("button", { name: /Commit changes/i }).click();
const dialog = page.getByRole("dialog");
await dialog.getByRole("textbox", { name: /Commit message/i }).fill("Add isolated cart precision review fixture");
await dialog.getByRole("radio", { name: /Create a new branch/i }).check();
const branch = dialog.getByRole("textbox", { name: "PR target branch", exact: true });
await branch.fill(input.branch);
await expect(branch).toHaveValue(input.branch);
await dialog.getByRole("button", { name: /Propose changes|Commit changes/i }).click();
await page.getByRole("textbox", { name: /Add a title/i }).fill(input.title);
await page.getByRole("textbox", { name: /Add a description|Comment|Description/i }).fill("Disposable automated review fixture. This money-total helper promises exact cent totals. Review only; do not modify or merge the branch. No manual bot mention is used.");
await page.getByRole("button", { name: "Create pull request", exact: true }).click();
await page.waitForURL(url => url.origin === "https://github.com" && /\/pull\/\d+$/.test(url.pathname));
const match = /\/pull\/(\d+)$/.exec(new URL(page.url()).pathname);
if (!match) throw new ConnectionFailure("created_pr_url_missing");
return Number(match[1]);
}
export async function observeGitHubHead(page: Page) {
const pull = new URL(page.url());
const repository = /^\/([^/]+\/[^/]+)\/pull\/\d+$/.exec(pull.pathname)?.[1];
if (pull.origin !== "https://github.com" || !repository) throw new ConnectionFailure("github_pr_evidence_missing");
const commits = page.getByRole("link", { name: /^Commits(?:\s*\(?\d+\)?)?$/ });
await expect(commits).toBeVisible();
const commitsUrl = new URL(await commits.getAttribute("href") ?? "", pull.origin);
if (commitsUrl.origin !== pull.origin || commitsUrl.pathname !== `${pull.pathname}/commits`) throw new ConnectionFailure("github_commits_navigation_missing");
// Follow the rendered navigation link. A click immediately after PR creation
// can be swallowed when GitHub replaces its server-rendered navigation.
await page.goto(commitsUrl.href);
const commit = page.getByRole("link", { name: /^[a-f0-9]{7,40}$/ }).last();
await expect(commit).toBeVisible();
const url = new URL(await commit.getAttribute("href") ?? "", pull.origin);
const sha = /\/(?:commits?|changes)\/([a-f0-9]{40})$/.exec(url.pathname)?.[1];
if (url.origin !== pull.origin || !sha || ![`/${repository}/commit/${sha}`, `${pull.pathname}/commits/${sha}`, `${pull.pathname}/changes/${sha}`].includes(url.pathname)) throw new ConnectionFailure("github_head_evidence_missing");
const conversation = page.getByRole("link", { name: /^Conversation(?:\s*\(?\d+\)?)?$/ });
await expect(conversation).toBeVisible();
const conversationUrl = new URL(await conversation.getAttribute("href") ?? "", pull.origin);
if (conversationUrl.origin !== pull.origin || conversationUrl.pathname !== pull.pathname) throw new ConnectionFailure("github_conversation_navigation_missing");
await page.goto(conversationUrl.href);
return sha;
}
export async function closeOwnedGitHubPullRequest(page: Page) {
const closed = page.getByText("Closed", { exact: true }).first();
if (!await closed.isVisible()) {
await expect(page.getByRole("button", { name: /^(?:Merge status|Status)$/ })).toBeEnabled();
await page.getByRole("button", { name: "Close pull request", exact: true }).click();
}
await expect(closed).toBeVisible({ timeout: 30_000 });
const restored = page.getByRole("button", { name: "Restore branch", exact: true });
if (!await restored.isVisible()) await page.getByRole("button", { name: "Delete branch", exact: true }).click();
await expect(restored).toBeVisible({ timeout: 30_000 });
}