mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 02:26:12 +02:00
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E checks verify real user paths through those agents. > - A recorded GitHub review-bot walkthrough had no repeatable registered case. > - App setup, automatic triggering, publication, and task completion need separate evidence. > - This pull request adds an explicit attended journey to the existing catalog and report tools. > - Reusable setup profiles and independent assertions preserve the same outcome across compatible targets. ## Linked Issues or Issue Description Refs: #17262. This case remains an explicit attended attached-target flow. It does not replace the shared deployment-target or paid-operations framework. **What existing behavior does this improve?** The Product E2E catalog can now select and retain evidence for GitHub review-bot setup and its first automatic review. **Current behavior** Recorded walkthroughs cannot be rerun as a fixed registered case. A published review alone can hide a failed run or unfinished task. **Proposed behavior** Create and install an owned App, save automatic new-PR settings, create a defective PR, and require correlated review delivery, an inline finding, a consistent check, a successful run, and a Done task. Retain failures and confirm owned-resource cleanup. **Reason and benefit** The case makes a useful external integration reproducible without treating setup, unit checks, or report rendering as proof of a completed live journey. ## What Changed - Add two explicit attended cells for an existing configured agent in local or Daytona execution. - Declare hosting, account, connection, and credential prerequisites through private setup bindings. - Add fixed browser controls, ownership checks, independent grading, and cleanup. - Package declared screenshots through the existing sanitizer, manifest, validator, and dashboard. - Reject operations-managed execution until budgeted routing is qualified. - Document hosted QA credential setup and evidence limits. ## Verification - Focused catalog, oracle, report, connection, and ownership checks: 127 tests passed. - Inherited-environment preflight regression failed against the previous commit and passed after the fix. Actual-run environment verification remains required. - Harness TypeScript check and full workspace typecheck passed. Full build passed. The local full-test attempt was interrupted after repeated embedded PostgreSQL startup failures; its exit code of zero does not establish a pass; Exact-head Linux CI passed the full test, typecheck, build, Runner, serialized, browser, and canary gates on `abbddeade848edefc8de75f11cc9008b2cd52764`. - A real hosted attempt published a rated review and inline finding. It failed when its native session timed out before the task reached Done. Cleanup passed. Full live qualification remains open. - Credential scan found no known credential values in the authored files or retained report corpus. ## Risks - The full case creates external test resources and uses the configured provider. It requires an explicit selection and current task authority. It is excluded from default and CI matrices. - Time and run limits are not a hard dollar cap. Missing dollar cost stays unknown. Fixture setup does not weaken budget policy. - Fresh personal OAuth, unattended authentication, runtime skill-read proof, and cross-host qualification are separate claims. - The selected live fixture needs environment/lifecycle recovery before it can adopt a longer provider deadline. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, code execution, and browser tools. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the enhancement template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
182 lines
12 KiB
TypeScript
182 lines
12 KiB
TypeScript
import { expect, type Page } from "@playwright/test";
|
|
import { ConnectionBlock, ConnectionFailure } from "./connection-config.js";
|
|
import { connectionDelay } from "./connection-target.js";
|
|
import type { GitHubReviewJourneyConfig } from "./github-review-config.js";
|
|
|
|
export async function enterGitHubJourneyCompany(page: Page, origin: string, company: { issuePrefix: string }) {
|
|
if (!/^[A-Za-z0-9]+$/.test(company.issuePrefix)) throw new ConnectionFailure("invalid_journey_company_prefix");
|
|
const pathname = `/${company.issuePrefix}/dashboard`;
|
|
await page.goto(`${origin}${pathname}`);
|
|
await page.waitForURL(url => url.origin === origin && url.pathname === pathname);
|
|
}
|
|
|
|
/** Capture measured pages only, excluding authentication and unrelated GitHub data. */
|
|
export function isGitHubJourneyEvidencePage(value: string, targetOrigin: string, repository: string, appSlug?: string) {
|
|
const url = new URL(value);
|
|
if ([...url.searchParams.keys()].some(key => /token|secret|code|state|password/i.test(key))) return false;
|
|
if (url.origin === targetOrigin) return !/connect|login|auth/.test(url.pathname);
|
|
if (url.origin !== "https://github.com") return false;
|
|
const prefix = `/${repository}/`;
|
|
if (url.pathname.startsWith(prefix) && /^(?:pull\/\d+(?:\/(?:files|commits|checks))?|runs\/\d+|actions\/runs\/\d+(?:\/job\/\d+)?)\/?$/.test(url.pathname.slice(prefix.length))) return true;
|
|
return Boolean(appSlug && (url.pathname === `/apps/${appSlug}/installations/new` || url.pathname === `/apps/${appSlug}/installations/new/permissions`));
|
|
}
|
|
|
|
export async function awaitJourneyCondition(check: () => Promise<boolean>, timeoutMs: number, code: string, signal: AbortSignal, boundary: "awaiting_user" | "failed" = "awaiting_user") {
|
|
const end = Date.now() + timeoutMs;
|
|
while (Date.now() < end) {
|
|
if (signal.aborted) throw new ConnectionBlock("blocked_target", "journey_interrupted");
|
|
if (await check()) return;
|
|
await connectionDelay(750);
|
|
}
|
|
if (boundary === "failed") throw new ConnectionFailure(code);
|
|
throw new ConnectionBlock("awaiting_user", code);
|
|
}
|
|
|
|
export async function observeGitHubJourneyAccount(page: Page) {
|
|
const menu = page.getByRole("button", { name: /Open user navigation menu|View profile and more/i });
|
|
const dialog = page.getByRole("dialog", { name: "User navigation", exact: true });
|
|
await menu.click();
|
|
try { await dialog.waitFor({ timeout: 3_000 }); }
|
|
catch (error) {
|
|
// GitHub can replace its server-rendered header during hydration. Retry
|
|
// once only when the observed control still explicitly says Open.
|
|
if (!await menu.isVisible() || await dialog.isVisible()) throw error;
|
|
await menu.click();
|
|
await dialog.waitFor();
|
|
}
|
|
const profile = dialog.getByRole("link", { name: "Profile", exact: true });
|
|
const href = await profile.getAttribute("href");
|
|
const url = new URL(href ?? "", "https://github.com");
|
|
const account = url.origin === "https://github.com" ? /^\/([A-Za-z0-9-]+)\/?$/.exec(url.pathname)?.[1] : undefined;
|
|
if (!account) throw new ConnectionFailure("github_profile_identity_missing");
|
|
await page.keyboard.press("Escape");
|
|
return account;
|
|
}
|
|
|
|
/** Prepare one repository grant; submitting Install remains a separate action. */
|
|
export async function selectGitHubJourneyRepository(page: Page, repository: string) {
|
|
const selectedOnly = page.getByRole("radio", { name: /Only select repositories/i });
|
|
await selectedOnly.check();
|
|
await page.getByRole("button", { name: /Select repositories/i }).click();
|
|
await page.getByRole("textbox", { name: "Search for a repository", exact: true }).fill(repository.split("/")[1]!);
|
|
const name = new RegExp(`^${repository.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}(?:\\s|$)`);
|
|
// GitHub nests the selectable repository item inside a containing menu item.
|
|
await page.getByRole("menuitem", { name }).last().click();
|
|
await page.keyboard.press("Escape");
|
|
await expect(selectedOnly).toBeChecked();
|
|
await expect(page.getByRole("radio", { name: /^All repositories/ })).not.toBeChecked();
|
|
await expect(page.getByRole("main")).toContainText("Selected 1 repository.");
|
|
await expect(page.getByRole("button", { name: `Remove ${repository.split("/")[1]}`, exact: true })).toBeVisible();
|
|
await expect(page.getByRole("button", { name: "Install", exact: true })).toBeEnabled();
|
|
}
|
|
|
|
/** Login remains ordinary browser UI. Passkeys are never exported or emulated. */
|
|
export async function ensureGitHubJourneyLogin(input: {
|
|
page: Page; config: GitHubReviewJourneyConfig; username: string; password?: () => Promise<string>; signal: AbortSignal;
|
|
checkpoint(id: string): Promise<void>;
|
|
}) {
|
|
const { page, config } = input;
|
|
const signedIn = () => page.getByRole("button", { name: /Open user navigation menu|View profile and more/i }).isVisible();
|
|
if (!await signedIn()) {
|
|
const loginLink = page.getByRole("link", { name: /^Sign in$/ });
|
|
if (await loginLink.isVisible()) await loginLink.click();
|
|
if (config.github.login === "password") {
|
|
await page.getByLabel("Username or email address", { exact: true }).fill(input.username);
|
|
await page.getByLabel("Password", { exact: true }).fill(await input.password!());
|
|
await page.getByRole("button", { name: "Sign in", exact: true }).click();
|
|
}
|
|
await input.checkpoint("awaiting_github_sign_in");
|
|
console.log("[github-review-journey] Complete Ralph QA GitHub sign-in/passkey in the dedicated QA browser. No credential needs to be pasted into chat.");
|
|
await awaitJourneyCondition(signedIn, config.browser.loginTimeoutMs, "github_login_required", input.signal);
|
|
}
|
|
// The login identifier can be an email address; verify the GitHub handle.
|
|
if ((await observeGitHubJourneyAccount(page)).toLowerCase() !== config.github.account.toLowerCase()) throw new ConnectionBlock("blocked_target", "github_account_mismatch");
|
|
}
|
|
|
|
/** Explicit one-time bootstrap. Later attempts reuse this private repository. */
|
|
export async function createPrivateJourneyRepository(page: Page, repository: string) {
|
|
const [owner, name] = repository.split("/");
|
|
await page.getByRole("button", { name: /Create new/i }).click();
|
|
await page.getByRole("menuitem", { name: /New repository/i }).click();
|
|
const ownerButton = page.getByRole("button", { name: /Owner|Choose an owner/i });
|
|
await expect(ownerButton).toHaveText(owner!);
|
|
await page.getByRole("textbox", { name: /Repository name/i }).fill(name!);
|
|
await expect(page.getByText(`${name} is available.`, { exact: true })).toBeVisible();
|
|
const privateChoice = page.getByRole("radio", { name: /Private/ });
|
|
if (await privateChoice.isVisible()) await privateChoice.check();
|
|
else {
|
|
await page.getByRole("button", { name: /Choose visibility|Public/i }).click();
|
|
await page.getByRole("menuitemradio", { name: /Private/ }).click();
|
|
}
|
|
await expect(page.getByRole("button", { name: "Choose visibility: Private", exact: true })).toBeVisible();
|
|
const readme = page.getByRole("button", { name: "Add README", exact: true });
|
|
if (await readme.getAttribute("aria-pressed") !== "true") await readme.click();
|
|
await expect(readme).toHaveAttribute("aria-pressed", "true");
|
|
await page.getByRole("button", { name: "Create repository", exact: true }).click();
|
|
await page.waitForURL(url => url.origin === "https://github.com" && url.pathname === `/${repository}`);
|
|
await expect(page.getByRole("link", { name: name!, exact: true }).first()).toBeVisible();
|
|
await expect(page.getByText("Private", { exact: true }).first()).toBeVisible();
|
|
}
|
|
|
|
export async function createGitHubJourneyPullRequest(input: { page: Page; branch: string; title: string; contents: string }) {
|
|
const { page } = input;
|
|
// The initial file list is server-rendered before GitHub's controls hydrate.
|
|
// Its loaded commit receipt provides a visible readiness signal.
|
|
await expect(page.getByRole("link", { name: /^Commit [a-f0-9]{7,40}$/ })).toBeVisible();
|
|
await page.getByRole("button", { name: "Add file", exact: true }).click();
|
|
await page.getByRole("menuitem", { name: "Create new file", exact: true }).click();
|
|
await page.getByRole("textbox", { name: /Name your file|File name/i }).fill("cart-total.js");
|
|
await page.getByRole("textbox", { name: /Editing cart-total.js file contents|File contents/i }).fill(input.contents);
|
|
await page.getByRole("button", { name: /Commit changes/i }).click();
|
|
const dialog = page.getByRole("dialog");
|
|
await dialog.getByRole("textbox", { name: /Commit message/i }).fill("Add isolated cart precision review fixture");
|
|
await dialog.getByRole("radio", { name: /Create a new branch/i }).check();
|
|
const branch = dialog.getByRole("textbox", { name: "PR target branch", exact: true });
|
|
await branch.fill(input.branch);
|
|
await expect(branch).toHaveValue(input.branch);
|
|
await dialog.getByRole("button", { name: /Propose changes|Commit changes/i }).click();
|
|
await page.getByRole("textbox", { name: /Add a title/i }).fill(input.title);
|
|
await page.getByRole("textbox", { name: /Add a description|Comment|Description/i }).fill("Disposable automated review fixture. This money-total helper promises exact cent totals. Review only; do not modify or merge the branch. No manual bot mention is used.");
|
|
await page.getByRole("button", { name: "Create pull request", exact: true }).click();
|
|
await page.waitForURL(url => url.origin === "https://github.com" && /\/pull\/\d+$/.test(url.pathname));
|
|
const match = /\/pull\/(\d+)$/.exec(new URL(page.url()).pathname);
|
|
if (!match) throw new ConnectionFailure("created_pr_url_missing");
|
|
return Number(match[1]);
|
|
}
|
|
|
|
export async function observeGitHubHead(page: Page) {
|
|
const pull = new URL(page.url());
|
|
const repository = /^\/([^/]+\/[^/]+)\/pull\/\d+$/.exec(pull.pathname)?.[1];
|
|
if (pull.origin !== "https://github.com" || !repository) throw new ConnectionFailure("github_pr_evidence_missing");
|
|
const commits = page.getByRole("link", { name: /^Commits(?:\s*\(?\d+\)?)?$/ });
|
|
await expect(commits).toBeVisible();
|
|
const commitsUrl = new URL(await commits.getAttribute("href") ?? "", pull.origin);
|
|
if (commitsUrl.origin !== pull.origin || commitsUrl.pathname !== `${pull.pathname}/commits`) throw new ConnectionFailure("github_commits_navigation_missing");
|
|
// Follow the rendered navigation link. A click immediately after PR creation
|
|
// can be swallowed when GitHub replaces its server-rendered navigation.
|
|
await page.goto(commitsUrl.href);
|
|
const commit = page.getByRole("link", { name: /^[a-f0-9]{7,40}$/ }).last();
|
|
await expect(commit).toBeVisible();
|
|
const url = new URL(await commit.getAttribute("href") ?? "", pull.origin);
|
|
const sha = /\/(?:commits?|changes)\/([a-f0-9]{40})$/.exec(url.pathname)?.[1];
|
|
if (url.origin !== pull.origin || !sha || ![`/${repository}/commit/${sha}`, `${pull.pathname}/commits/${sha}`, `${pull.pathname}/changes/${sha}`].includes(url.pathname)) throw new ConnectionFailure("github_head_evidence_missing");
|
|
const conversation = page.getByRole("link", { name: /^Conversation(?:\s*\(?\d+\)?)?$/ });
|
|
await expect(conversation).toBeVisible();
|
|
const conversationUrl = new URL(await conversation.getAttribute("href") ?? "", pull.origin);
|
|
if (conversationUrl.origin !== pull.origin || conversationUrl.pathname !== pull.pathname) throw new ConnectionFailure("github_conversation_navigation_missing");
|
|
await page.goto(conversationUrl.href);
|
|
return sha;
|
|
}
|
|
|
|
export async function closeOwnedGitHubPullRequest(page: Page) {
|
|
const closed = page.getByText("Closed", { exact: true }).first();
|
|
if (!await closed.isVisible()) {
|
|
await expect(page.getByRole("button", { name: /^(?:Merge status|Status)$/ })).toBeEnabled();
|
|
await page.getByRole("button", { name: "Close pull request", exact: true }).click();
|
|
}
|
|
await expect(closed).toBeVisible({ timeout: 30_000 });
|
|
const restored = page.getByRole("button", { name: "Restore branch", exact: true });
|
|
if (!await restored.isVisible()) await page.getByRole("button", { name: "Delete branch", exact: true }).click();
|
|
await expect(restored).toBeVisible({ timeout: 30_000 });
|
|
}
|