Files
PaperClipAI/tests/runner-e2e/github-review-config.ts
T
DottaandPaperclip 6d46851920 test(e2e): add attended GitHub review bot journey (#17457)
## Thinking Path

> - Paperclip manages agents and the work they perform.
> - Product E2E checks verify real user paths through those agents.
> - A recorded GitHub review-bot walkthrough had no repeatable
registered case.
> - App setup, automatic triggering, publication, and task completion
need separate evidence.
> - This pull request adds an explicit attended journey to the existing
catalog and report tools.
> - Reusable setup profiles and independent assertions preserve the same
outcome across compatible targets.

## Linked Issues or Issue Description

Refs: #17262. This case remains an explicit attended attached-target
flow. It does not replace the shared deployment-target or
paid-operations framework.

**What existing behavior does this improve?**

The Product E2E catalog can now select and retain evidence for GitHub
review-bot setup and its first automatic review.

**Current behavior**

Recorded walkthroughs cannot be rerun as a fixed registered case. A
published review alone can hide a failed run or unfinished task.

**Proposed behavior**

Create and install an owned App, save automatic new-PR settings, create
a defective PR, and require correlated review delivery, an inline
finding, a consistent check, a successful run, and a Done task. Retain
failures and confirm owned-resource cleanup.

**Reason and benefit**

The case makes a useful external integration reproducible without
treating setup, unit checks, or report rendering as proof of a completed
live journey.

## What Changed

- Add two explicit attended cells for an existing configured agent in
local or Daytona execution.
- Declare hosting, account, connection, and credential prerequisites
through private setup bindings.
- Add fixed browser controls, ownership checks, independent grading, and
cleanup.
- Package declared screenshots through the existing sanitizer, manifest,
validator, and dashboard.
- Reject operations-managed execution until budgeted routing is
qualified.
- Document hosted QA credential setup and evidence limits.

## Verification

- Focused catalog, oracle, report, connection, and ownership checks: 127
tests passed.
- Inherited-environment preflight regression failed against the previous
commit and passed after the fix. Actual-run environment verification
remains required.
- Harness TypeScript check and full workspace typecheck passed. Full
build passed. The local full-test attempt was interrupted after repeated
embedded PostgreSQL startup failures; its exit code of zero does not
establish a pass; Exact-head Linux CI passed the full test, typecheck,
build, Runner, serialized, browser, and canary gates on
`abbddeade848edefc8de75f11cc9008b2cd52764`.
- A real hosted attempt published a rated review and inline finding. It
failed when its native session timed out before the task reached Done.
Cleanup passed. Full live qualification remains open.
- Credential scan found no known credential values in the authored files
or retained report corpus.

## Risks

- The full case creates external test resources and uses the configured
provider. It requires an explicit selection and current task authority.
It is excluded from default and CI matrices.
- Time and run limits are not a hard dollar cap. Missing dollar cost
stays unknown. Fixture setup does not weaken budget policy.
- Fresh personal OAuth, unattended authentication, runtime skill-read
proof, and cross-host qualification are separate claims.
- The selected live fixture needs environment/lifecycle recovery before
it can adopt a longer provider deadline.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, code execution, and
browser tools. The exact deployment model ID and context-window size are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, including the available version
and capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the enhancement
template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open findings
- [x] I will address all Greptile and reviewer comments before merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 12:16:26 -07:00

70 lines
5.6 KiB
TypeScript

import { readFile } from "node:fs/promises";
import path from "node:path";
import os from "node:os";
import { ConnectionBlock, parseConnectionConfig, targetOrigin, type ConnectionConfig } from "./connection-config.js";
import { parseJourneyCredentialBinding, type JourneyCredentialBinding } from "./journey-dependencies.js";
export interface GitHubReviewJourneyConfig {
version: 1;
setupId: string;
hosting: "self-hosted" | "paperclip-cloud";
executionEnvironment: "local" | "daytona";
target: Extract<ConnectionConfig["target"], { mode: "attach" }> & { companyId: string; environmentId: string };
browser: ConnectionConfig["browser"];
authentication?: { module: string; configFile: string };
agent: { id: string; name: string; skill: string; model: string; provider: string; runtimeMode: "native" | "legacy" };
github: { repository: string; ownerType: "personal" | "organization"; appOwner: string; account: string; login: "passkey" | "password"; username: JourneyCredentialBinding; password?: JourneyCredentialBinding };
deadlineMs: number;
maximumRuns: number;
}
const invalid = (): never => { throw new ConnectionBlock("blocked_target", "invalid_github_journey_configuration"); };
function object(v: unknown, keys: string[]): Record<string, any> {
if (!v || typeof v !== "object" || Array.isArray(v) || Object.keys(v).some(k => !keys.includes(k))) return invalid();
return v as Record<string, any>;
}
function text(v: unknown, pattern?: RegExp): string {
if (typeof v !== "string" || !v.trim() || v.length > 1024 || (pattern && !pattern.test(v))) return invalid();
return v;
}
export function parseGitHubReviewJourneyConfig(input: unknown): GitHubReviewJourneyConfig {
try {
const v = object(input, ["version", "setupId", "hosting", "executionEnvironment", "target", "browser", "authentication", "agent", "github", "deadlineMs", "maximumRuns"]);
if (v.version !== 1 || !["self-hosted", "paperclip-cloud"].includes(v.hosting)) return invalid();
if (v.executionEnvironment !== undefined && !["local", "daytona"].includes(v.executionEnvironment)) return invalid();
const target = object(v.target, ["mode", "baseURL", "expectedCommit", "deploymentMode", "companyId", "environmentId"]);
if (target.mode !== "attach") return invalid();
const common = parseConnectionConfig({ target, browser: v.browser });
if (common.target.mode !== "attach" || !common.browser.profileDir) return invalid();
const agent = object(v.agent, ["id", "name", "skill", "model", "provider", "runtimeMode"]);
const github = object(v.github, ["repository", "ownerType", "appOwner", "account", "login", "username", "password"]);
if (!["personal", "organization"].includes(github.ownerType) || !["passkey", "password"].includes(github.login)
|| !["native", "legacy"].includes(agent.runtimeMode) || (github.login === "password" && !github.password)) return invalid();
const deadlineMs = v.deadlineMs ?? 900_000; const maximumRuns = v.maximumRuns ?? 3;
if (!Number.isInteger(deadlineMs) || deadlineMs < 60_000 || deadlineMs > 1_800_000
|| !Number.isInteger(maximumRuns) || maximumRuns < 1 || maximumRuns > 5) return invalid();
const uuid = /^[a-f0-9]{8}-(?:[a-f0-9]{4}-){3}[a-f0-9]{12}$/;
const authentication = v.authentication === undefined ? undefined : object(v.authentication, ["module", "configFile"]);
if (authentication && (!path.isAbsolute(text(authentication.module)) || !path.isAbsolute(text(authentication.configFile)))) return invalid();
return { version: 1, setupId: text(v.setupId, /^[a-z0-9-]{1,80}$/), hosting: v.hosting, executionEnvironment: v.executionEnvironment ?? "local",
target: { ...common.target, baseURL: targetOrigin(common.target.baseURL), companyId: text(target.companyId, uuid), environmentId: text(target.environmentId, uuid) }, browser: common.browser,
...(authentication ? { authentication: { module: authentication.module, configFile: authentication.configFile } } : {}),
agent: { id: text(agent.id, uuid), name: text(agent.name), skill: text(agent.skill, /^[a-z0-9-]+$/), model: text(agent.model), provider: text(agent.provider), runtimeMode: agent.runtimeMode },
github: { repository: text(github.repository, /^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/), ownerType: github.ownerType, appOwner: text(github.appOwner, /^[A-Za-z0-9-]+$/),
account: text(github.account ?? (github.ownerType === "personal" ? github.appOwner : undefined), /^[A-Za-z0-9-]+$/), login: github.login,
username: parseJourneyCredentialBinding(github.username), ...(github.password ? { password: parseJourneyCredentialBinding(github.password) } : {}) }, deadlineMs, maximumRuns };
} catch { return invalid(); }
}
export async function loadGitHubReviewJourneyConfig(file?: string) {
if (!file) throw new ConnectionBlock("blocked_target", "github_journey_config_required");
try {
const value = JSON.parse(await readFile(file, "utf8"));
if (value.browser?.profileDir?.startsWith("~/")) value.browser.profileDir = path.join(os.homedir(), value.browser.profileDir.slice(2));
// Profile paths are relative to the profile, not the invoking checkout.
if (value.browser?.profileDir && !path.isAbsolute(value.browser.profileDir)) value.browser.profileDir = path.resolve(path.dirname(file), value.browser.profileDir);
for (const key of ["module", "configFile"]) if (value.authentication?.[key] && !path.isAbsolute(value.authentication[key])) value.authentication[key] = path.resolve(path.dirname(file), value.authentication[key]);
return parseGitHubReviewJourneyConfig(value);
} catch { throw new ConnectionBlock("blocked_target", "invalid_github_journey_configuration"); }
}