DottaandPaperclip
6d46851920
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E checks verify real user paths through those agents. > - A recorded GitHub review-bot walkthrough had no repeatable registered case. > - App setup, automatic triggering, publication, and task completion need separate evidence. > - This pull request adds an explicit attended journey to the existing catalog and report tools. > - Reusable setup profiles and independent assertions preserve the same outcome across compatible targets. ## Linked Issues or Issue Description Refs: #17262. This case remains an explicit attended attached-target flow. It does not replace the shared deployment-target or paid-operations framework. **What existing behavior does this improve?** The Product E2E catalog can now select and retain evidence for GitHub review-bot setup and its first automatic review. **Current behavior** Recorded walkthroughs cannot be rerun as a fixed registered case. A published review alone can hide a failed run or unfinished task. **Proposed behavior** Create and install an owned App, save automatic new-PR settings, create a defective PR, and require correlated review delivery, an inline finding, a consistent check, a successful run, and a Done task. Retain failures and confirm owned-resource cleanup. **Reason and benefit** The case makes a useful external integration reproducible without treating setup, unit checks, or report rendering as proof of a completed live journey. ## What Changed - Add two explicit attended cells for an existing configured agent in local or Daytona execution. - Declare hosting, account, connection, and credential prerequisites through private setup bindings. - Add fixed browser controls, ownership checks, independent grading, and cleanup. - Package declared screenshots through the existing sanitizer, manifest, validator, and dashboard. - Reject operations-managed execution until budgeted routing is qualified. - Document hosted QA credential setup and evidence limits. ## Verification - Focused catalog, oracle, report, connection, and ownership checks: 127 tests passed. - Inherited-environment preflight regression failed against the previous commit and passed after the fix. Actual-run environment verification remains required. - Harness TypeScript check and full workspace typecheck passed. Full build passed. The local full-test attempt was interrupted after repeated embedded PostgreSQL startup failures; its exit code of zero does not establish a pass; Exact-head Linux CI passed the full test, typecheck, build, Runner, serialized, browser, and canary gates on `abbddeade848edefc8de75f11cc9008b2cd52764`. - A real hosted attempt published a rated review and inline finding. It failed when its native session timed out before the task reached Done. Cleanup passed. Full live qualification remains open. - Credential scan found no known credential values in the authored files or retained report corpus. ## Risks - The full case creates external test resources and uses the configured provider. It requires an explicit selection and current task authority. It is excluded from default and CI matrices. - Time and run limits are not a hard dollar cap. Missing dollar cost stays unknown. Fixture setup does not weaken budget policy. - Fresh personal OAuth, unattended authentication, runtime skill-read proof, and cross-host qualification are separate claims. - The selected live fixture needs environment/lifecycle recovery before it can adopt a longer provider deadline. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, code execution, and browser tools. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the enhancement template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
7.4 KiB
GitHub review bot journey (explicit, attended)
The github-review-journey suite converts the recorded Ponytail walkthrough into
fixed browser actions and an independent review oracle. It creates a GitHub App,
installs it on one private eval repository, saves automatic review configuration
with a selected skill and 5/5 passing threshold, opens a defective PR without a
bot mention, and checks the matching automatic task, review, GitHub delivery,
failed check, task completion, and persistence.
The suite is applicable to self-hosted deployments with public GitHub webhook
ingress and to Paperclip Cloud with GitHub App support. Hosting is independent of
the configured agent's local or daytona execution environment. Both cells are
registered; select exactly one and bind it in the private setup profile. Browser
registration and oracle tests do not establish live qualification of either cell.
Dependencies and starting state
Use a dedicated private browser profile outside the checkout. Supply the intended company, serving SHA, deployment mode, existing agent/model/provider/runtime, review skill, agent environment ID, GitHub account handle, App owner, and private reusable repository. The GitHub login identifier may be an email; the handle is separate. The agent and its provider connection must already be configured. A matching existing GitHub user connection is required; fresh user OAuth and agent hiring are outside this variant. App creation and installation remain measured UI actions with attended consent/passkey boundaries.
The browser, harness, and review agent need no external MCP tools. The agent uses the task-bound GitHub review tools. The preexisting personal GitHub connection is a separate product identity prerequisite; the product may store/expose it as an MCP-backed connection. That does not supply MCP access to the harness or review agent. The browser account controls the App and eval repository; the agent consumes its own provider and task bindings. Harness access to 1Password does not supply those product connections.
Private setup profiles bind credential aliases to environment variable names or
selected op://vault/item/field references. A 1Password service-account token
goes only to the resolver's op child, and browser processes inherit a limited
environment. Never commit passwords, mailbox keys, passkeys, cookies, auth traces,
or browser profile contents. A private optional authentication module and
config file can prepare an ordinary hosted session before measured actions;
the private QA implementation reuses Cloud's email-code authentication helper.
Select and run
pnpm test:e2e:runner -- --list --suite github-review-journey
pnpm test:e2e:runner -- \
--id github-review-journey.configured-agent.daytona.setup-and-first-review \
--github-review-config /private/ponytail-qa.json
The private definition, reference-only setup example, credential setup utilities,
and source-step mapping live in paperclip-evals/evals/product-journeys/ponytail.
Its wrapper supports the host's Keychain bootstrap without putting a token in
shell arguments. The suite is excluded from --all and generated CI matrices.
It has no scheduler, public upload, or automatic retry. A full setup needs
attended external authorization. Do not claim it is an unattended CI journey.
In a terminal, each creation/installation pause also accepts the exact displayed
action and unique App name. Inspect that attempt's permissions first. The setup
profile is not authorization: use the current task's actual authority, including
any explicitly granted standing permission for disposable QA resources. Do not
repeat an already covered approval; obey any required host approval boundary.
Oracle and ownership
cart-total.js is a newly authored deterministic precision defect, not an
assertion that the source walkthrough had identical fixture bytes. A passing
eval requires a complete assessment of that path and head, a relevant inline
precision finding, a score below 5, and a correctly failed GitHub check. The
oracle rejects a stale/foreign company, endpoint, issue, PR, or head, manual
triggering, missing findings/delivery, and false completion. It does not require
the source's exact 3/5 score. Persisted skill selection proves binding; this
variant does not yet independently verify a runtime skill-read receipt.
The rendered inline discussion must identify the created bot, cart-total.js,
the complete reviewed head, and the relevant precision finding. A summary that
mentions the defect does not substitute for inline publication. The review's
completed state alone is also insufficient: the actual run must succeed and the
associated task must reach Done. A native failure after publication fails the
journey and retains the publication evidence already observed.
Each attempt records endpoint, App, PR/branch, task, and run ownership. Cleanup removes the owned endpoint, closes the owned PR/deletes its branch, and asks for the owned App's removal. App identity must match the created endpoint, company, agent, exact name, GitHub-confirmed owner/slug, and immutable App ID. Installation management is recorded separately: uninstalling alone does not prove App deletion. It retains the shared repository, account, provider/user connections, and completed task evidence. An unconfirmed cleanup fails the attempt; retain the resource receipt for recovery. Read the receipt before any manual cleanup. Never delete by a guessed name or broad account search.
The configured deadline and observed run limit bound the attempt. They are not
a hard provider-dollar cap or proof that no request outlives cancellation. Use
the existing budgeted Cloud operations adapter before automated paid campaigns;
this attended attached-agent variant does not replace that infrastructure.
Effective company/agent budget policies are recorded when readable. A route
that reports tokens without a dollar cost can stop on unpricedUsagePolicy: block despite a zero legacy spend counter. Any authorized exception is a
distinct tested setup condition; normal fixture setup must not silently grant
it. Restore temporary exceptions after verification and retain their receipts.
The case declares a ten-minute provider-turn allowance within its fifteen-minute body deadline. Read back the attached agent's effective timeout separately; catalog metadata cannot increase a deployed agent's limit. If the target rejects the required configuration or its lifecycle cannot settle, preserve that setup failure and repair the fixture before claiming a qualified run.
Private normalized results and the existing Product E2E dashboard retain actual model/provider, selected environment, serving SHA, setup ID, source hash, checkpoints, screenshots, observed usage, partial cost coverage, assistance, and cleanup. Missing usage remains unknown. Authentication screens are excluded.
github-review-report.ts uses the existing evidence sanitizer and manifest,
stages only declared screenshots, and verifies screenshot hashes and attempt
path ownership. The standard report inputs are evidence/result.json,
evidence/evidence-manifest.json, and evidence/snapshots/. Raw private operator
receipts are outside the manifest. Existing report tooling imports these files;
no second runner or report service is required. A presentation repair must use a
new directory, preserve the original result/hash and failure, and dispatch no
provider work. Rebuilding a dashboard never qualifies a failed live attempt.