mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
73 lines
17 KiB
TypeScript
73 lines
17 KiB
TypeScript
/** Standalone private required-human UI. Never runs inside Playwright Test and
|
|
* never automates or imitates the external Grok desktop application. */
|
|
import { lstat, readFile } from "node:fs/promises";
|
|
import { chromium, test, type Browser, type BrowserContext, type Page } from "@playwright/test";
|
|
import { readValidatedPublicOriginReceipt } from "../chat-public-origin.js";
|
|
import { buildGrokAccessPrompt, buildGrokRoutinePrompt } from "../../../ui/src/lib/grok-bot-setup-prompt.js";
|
|
import type { GrokConnection, GrokInvitation } from "../../../packages/shared/src/grok-bot.js";
|
|
import type { McpConnectionRequest, McpDotPairingPreview } from "../../../packages/shared/src/public-mcp.js";
|
|
import { grokOperatorInputMatches, grokConsentUrlAllowed, grokCallbackOriginAllowed, type GrokOperatorInput, type GrokOperatorReceipt } from "./runtime-grok-operator-contract.js";
|
|
const esc = (s: string) => s.replace(/[&<>"']/g, c => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c]!));
|
|
async function pane(page: Page, title: string, body: string, checks: string[], button: string) {
|
|
await page.setContent(`<html><head><title>Private Grok operator handoff</title></head><body><h1>${esc(title)}</h1>${body}<form id="handoff">${checks.map((text, n) => `<p><label><input type="checkbox" required id="check-${n}">${esc(text)}</label></p>`).join("")}<button type="submit">${esc(button)}</button></form><script>document.querySelector('#handoff').onsubmit=e=>{e.preventDefault();document.body.dataset.confirmed='yes'}</script></body></html>`); await page.bringToFront();
|
|
}
|
|
async function humanConfirm(page: Page, timeout: number) { await page.waitForFunction(() => document.body.dataset.confirmed === "yes", undefined, { timeout }); }
|
|
async function json<T>(context: BrowserContext, origin: string, pathname: string): Promise<T> { const response = await context.request.get(origin + pathname); if (!response.ok()) throw new Error("Actual public Grok fixture read rejected"); return response.json() as Promise<T>; }
|
|
async function generatedPrompt(page: Page, name: string) { await page.getByRole("button", { name, exact: true }).click(); const preview = page.getByLabel("Setup prompt", { exact: true }); await preview.waitFor(); const text = await preview.evaluate(el => el instanceof HTMLTextAreaElement ? el.value : el.textContent ?? ""); await page.getByRole("button", { name: "Close agent setup", exact: true }).click(); return text; }
|
|
async function main(): Promise<GrokOperatorReceipt> {
|
|
let attached = false; try { test.info(); attached = true; } catch { /* must remain standalone */ } if (attached) throw new Error("Grok operator must not execute inside Test");
|
|
const filename = process.argv[2]!, stat = await lstat(filename); if (!stat.isFile() || stat.isSymbolicLink() || stat.uid !== process.getuid?.() || stat.mode & 0o077 || stat.size > 2 * 1024 * 1024) throw new Error("Invalid private Grok operator input file");
|
|
const i = JSON.parse(await readFile(filename, "utf8")) as GrokOperatorInput; if (!grokOperatorInputMatches(i)) throw new Error("Invalid exact selected Grok operator input");
|
|
const origin = await readValidatedPublicOriginReceipt({ receiptPath: i.receiptPath, executionId: i.executionId, attemptId: i.attemptId, backendOrigin: i.backendOrigin }); if (origin.publicOrigin !== i.publicOrigin) throw new Error("Grok operator public origin changed");
|
|
let browser: Browser | undefined, context: BrowserContext | undefined, product: Page | undefined, pairingCode = "", callbackOrigin: string | undefined, privacyViolation = false; const ownedClipboard = new Set<string>(); if (i.operation === "setup") { ownedClipboard.add(i.webhookUrl!); ownedClipboard.add(i.webhookKey!); }
|
|
const deadline = setTimeout(() => { void browser?.close(); }, i.operation === "availability" ? 30000 : 360000); deadline.unref();
|
|
const stop = () => { void (async () => { try { await browser?.close(); } finally { process.exit(1); } })(); }; process.once("SIGTERM", stop);
|
|
try {
|
|
browser = await chromium.launch({ headless: false }); context = await browser.newContext({ storageState: i.publicSession }); context.setDefaultTimeout(15000); context.setDefaultNavigationTimeout(15000);
|
|
await context.grantPermissions(["clipboard-read", "clipboard-write"], { origin: i.publicOrigin });
|
|
await context.route("**/*", route => { const req = route.request(), u = new URL(req.url()); if (pairingCode && req.postData()?.includes(pairingCode) && u.origin !== i.publicOrigin) { privacyViolation = true; return route.abort(); } return u.origin === i.publicOrigin || callbackOrigin === u.origin && req.isNavigationRequest() ? route.continue() : route.abort(); });
|
|
const person = await json<{ user: { id: string; name: string } }>(context, i.publicOrigin, "/api/auth/get-session"), company = await json<{ id: string; name: string }>(context, i.publicOrigin, `/api/companies/${i.companyId}`);
|
|
if (person.user.id !== i.personId || person.user.name !== i.personName || company.id !== i.companyId || company.name !== i.companyName) throw new Error("Actual public fixture person/company did not match");
|
|
const operator = await context.newPage(), identity = `<p>Attempt ${esc(i.attemptId)} · Operator ${esc(i.operatorRef)}</p><p>Existing Bot: <strong>${esc(i.botName)}</strong> · authorization reference ${esc(i.botAuthorizationRef)}</p><p>Paperclip person ${esc(i.personName)} (${esc(i.personId)}), company ${esc(i.companyName)} (${esc(i.companyId)}).</p>`;
|
|
const common = { operation: i.operation, executionId: i.executionId, attemptId: i.attemptId, companyId: i.companyId, personId: i.personId, operatorRef: i.operatorRef, botAuthorizationRef: i.botAuthorizationRef, processIsolation: true, testRunnerAttached: false, humanSelectedAuthorizedBot: true, publicPersonMatched: true, privateStorageClean: true, clipboardClean: true } as const;
|
|
if (i.operation === "availability") {
|
|
await pane(operator, "Required Grok operator availability", identity + "<p>This is a required human prerequisite, before the canonical model request. No provider setup or consent has occurred. Presence must be confirmed within 15 seconds.</p>", ["I am the selected authorized operator of this exact existing Grok desktop Bot.", "I am available for this attempt and can use the real Bot and its OAuth browser flow; no substitute Bot or simulated receipt will be used."], "Confirm availability");
|
|
await humanConfirm(operator, 15000); return common;
|
|
}
|
|
const resumePath = `/api/companies/${i.companyId}/grok-invitations`, route = (id: string) => `/api/companies/${i.companyId}/agents/${id}/grok-binding`;
|
|
product = await context.newPage(); let invitation = await json<GrokInvitation | null>(context, i.publicOrigin, resumePath);
|
|
if (i.existingAgentId) { if (invitation?.agent.id !== i.existingAgentId || invitation.agent.botName !== i.botName || invitation.binding?.connected) throw new Error("Canonical existing invitation does not identify the chosen unconnected Bot"); await product.goto(`${i.publicOrigin}/${i.prefix}/agents/${i.existingAgentId}/runtime`); await product.getByRole("region", { name: "Grok Bot connection", exact: true }).getByRole("button", { name: "Reconnect agent access", exact: true }).click(); }
|
|
else {
|
|
if (invitation) throw new Error("An undeclared Grok invitation appeared before visible human creation"); await product.goto(`${i.publicOrigin}/${i.prefix}/agents`); await product.getByRole("button", { name: "New Agent", exact: true }).click(); await product.getByRole("button", { name: "Invite an external agent", exact: true }).click(); await product.getByRole("button", { name: /Grok Bot.*An existing Bot in Grok/ }).click();
|
|
const dialog = product.getByRole("dialog"); await dialog.getByLabel("Paperclip name", { exact: true }).fill("Existing Grok Bot"); await dialog.getByLabel("Bot name in Grok", { exact: true }).fill(i.botName); await dialog.getByLabel("Role", { exact: true }).selectOption("engineer"); await dialog.getByLabel("Manager", { exact: true }).selectOption(i.leadId); await dialog.getByRole("button", { name: "Continue", exact: true }).click();
|
|
}
|
|
const codeField = product.getByLabel("One-use pairing code", { exact: true }); await codeField.waitFor(); if (await codeField.getAttribute("type") !== "password") throw new Error("Actual generated pairing code is not protected"); pairingCode = await codeField.inputValue(); if (!/^[A-Za-z0-9_-]{32}$/.test(pairingCode)) throw new Error("No actual product-generated one-use pairing code"); ownedClipboard.add(pairingCode);
|
|
invitation = await json<GrokInvitation | null>(context, i.publicOrigin, resumePath); if (!invitation || invitation.agent.botName !== i.botName || invitation.approvalId || !invitation.binding || invitation.binding.connected) throw new Error("Actual visible Grok invitation is not the exact unconnected approved target");
|
|
const agentId = invitation.agent.id, bindingId = invitation.binding.id, initial = await json<GrokConnection>(context, i.publicOrigin, route(agentId)); if (!initial.resourceUrl || new URL(initial.resourceUrl).origin !== i.publicOrigin || initial.binding?.id !== bindingId) throw new Error("Product-generated access resource escaped the actual instance");
|
|
const access = await generatedPrompt(product, "Copy connection instructions"); if (![buildGrokAccessPrompt(initial.resourceUrl, i.botName), buildGrokAccessPrompt(initial.resourceUrl, "your Bot")].includes(access) || access.includes(pairingCode)) throw new Error("Actual access instructions differ from the production formatter or contain pairing authority"); ownedClipboard.add(access);
|
|
await pane(operator, "Send actual connection instructions to the selected Grok Bot", identity + `<p>In the real Grok desktop app, open this exact existing Bot and send ONLY these actual product-generated access instructions. Never send the pairing code to Bot chat.</p><textarea readonly rows="12">${esc(access)}</textarea><p>Let the Bot initiate its real custom MCP OAuth flow. Paste the resulting Paperclip consent-page URL below; this private fixture-person browser will open it. Do not paste a callback URL or credentials.</p><label>Actual Bot-initiated Paperclip consent page <input id="consent-url" type="url" required size="90"></label>`, ["I sent these actual instructions to the selected authorized existing Bot, and the URL comes from that Bot's actual OAuth flow."], "Open actual consent page");
|
|
await humanConfirm(operator, 150000); const consentUrl = await operator.locator("#consent-url").inputValue(); ownedClipboard.add(consentUrl); if (!grokConsentUrlAllowed(consentUrl, i.publicOrigin)) throw new Error("Operator supplied a foreign or unsupported consent URL");
|
|
const requestId = new URL(consentUrl).pathname.split("/").at(-1)!, requestPrefix = new URL(consentUrl).pathname.startsWith("/dot-connect/") ? "/api/dot-mcp/requests" : "/api/mcp/requests", request = await json<McpConnectionRequest>(context, i.publicOrigin, `${requestPrefix}/${requestId}`);
|
|
if (!request.agentConnection || request.requiresSignIn || !request.clientName?.trim() || !request.redirectOrigin || !grokCallbackOriginAllowed(request.redirectOrigin)) throw new Error("Actual external Bot request is not an authenticated supported agent OAuth request");
|
|
await pane(operator, "Review the actual external Bot OAuth client", identity + `<p>Actual request client: ${esc(request.clientName)}. Actual callback origin: ${esc(request.redirectOrigin)}.</p><p>Confirm this is the real selected Grok desktop Bot's pending connection. This approval only authorizes that exact callback origin for the real OAuth return.</p>`, ["I checked the real Bot's OAuth browser request and approve this exact client and callback origin."], "Review actual Paperclip consent"); await humanConfirm(operator, 60000); callbackOrigin = request.redirectOrigin;
|
|
const consent = await context.newPage(); await consent.goto(consentUrl); const previewPending = consent.waitForResponse(r => r.url() === `${i.publicOrigin}${requestPrefix}/${requestId}/dot-pairing/preview` && r.request().method() === "POST" && r.ok()); await consent.getByLabel("Pairing code", { exact: true }).fill(pairingCode); const previewResponse = await previewPending; const preview = await previewResponse.json() as McpDotPairingPreview;
|
|
if (preview.providerKind !== "grok_bot" || preview.company.id !== i.companyId || preview.agent.id !== agentId || preview.authorizingPerson !== i.personName) throw new Error("Actual consent preview selected another provider/company/agent/person");
|
|
const consentPending = consent.waitForResponse(r => r.url() === `${i.publicOrigin}${requestPrefix}/${requestId}/dot-pairing` && r.request().method() === "POST", { timeout: 120000 });
|
|
await pane(operator, "Approve only the reviewed real Paperclip form", identity + "<p>The real Paperclip OAuth page is open in the other tab with the private code entered. Review its Company, Agent and Authorized by fields. Click Allow agent connection THERE if correct. The code never goes to Bot chat. Then return to this operator tab.</p>", [], "I reviewed the real consent tab"); await consent.bringToFront();
|
|
const consentResponse = await consentPending;
|
|
if (!consentResponse.ok() || JSON.stringify(consentResponse.request().postDataJSON()) !== JSON.stringify({ pairingCode })) throw new Error("Actual human pairing consent did not submit the exact generated code");
|
|
const result = await consentResponse.json() as { redirectUrl?: string }; if (!result.redirectUrl || new URL(result.redirectUrl).origin !== callbackOrigin) throw new Error("Actual OAuth return escaped the operator-approved callback origin"); ownedClipboard.add(result.redirectUrl); for (const key of ["code", "state"]) { const value = new URL(result.redirectUrl).searchParams.get(key); if (value) ownedClipboard.add(value); }
|
|
const publicPerson = await json<{ user: { id: string } }>(context, i.publicOrigin, "/api/auth/get-session"); if (publicPerson.user.id !== i.personId) throw new Error("Public authorizing person changed during consent");
|
|
await product.goto(`${i.publicOrigin}/${i.prefix}/agents/${agentId}/runtime`); const section = product.getByRole("region", { name: "Grok Bot connection", exact: true }); await section.getByText(/Agent access connected/).waitFor(); await section.getByLabel("Bot name in Grok", { exact: true }).fill(i.botName);
|
|
const routine = await generatedPrompt(product, "Copy routine instructions"); if (routine !== buildGrokRoutinePrompt(bindingId, i.botName) || routine.includes(pairingCode) || routine.includes(i.webhookKey!)) throw new Error("Actual generated routine instructions are not secret-free and exactly target-bound"); ownedClipboard.add(routine);
|
|
await pane(operator, "Enable the real same-Bot routine", identity + `<p>Send these actual product-generated routine instructions to the SAME existing Bot. Reuse only the explicitly designated disposable routine; inspect its real Webhook panel and compare it with the selected credentials below. No model work or readiness is proven by this acknowledgement.</p><textarea readonly rows="12">${esc(routine)}</textarea><p>Selected webhook URL <input type="password" readonly value="${esc(i.webhookUrl!)}"></p><p>Selected webhook key <input type="password" readonly value="${esc(i.webhookKey!)}"></p>`, ["I sent these actual routine instructions to the same authorized Bot; I did not send the pairing code or Paperclip credentials to Bot chat.", "I inspected the actual designated routine's Webhook panel and its URL/key match these selected credentials; no unrelated routine was adopted."], "Record human handoff only"); await humanConfirm(operator, 120000);
|
|
if (privacyViolation) throw new Error("Pairing authority attempted to leave the actual public Paperclip form");
|
|
return { ...common, agentId, bindingId, productInvitationUi: true, accessInstructionsDelivered: true, routineInstructionsDelivered: true, publicPairingConsentSubmitted: true, publicCompanyAgentMatched: true, clientCallbackApproved: true, pairingCodeConfinedToPublicForm: true };
|
|
} finally {
|
|
clearTimeout(deadline); process.removeListener("SIGTERM", stop);
|
|
try { if (ownedClipboard.size && context) { const clipboardPage = product && !product.isClosed() ? product : await context.newPage(); if (new URL(clipboardPage.url()).origin !== i.publicOrigin) await clipboardPage.goto(i.publicOrigin); const current = await clipboardPage.evaluate(() => navigator.clipboard.readText()); if (ownedClipboard.has(current)) await clipboardPage.evaluate(() => navigator.clipboard.writeText("")); } }
|
|
finally { try { await context?.clearCookies(); } finally { try { await context?.close(); } finally { await browser?.close(); pairingCode = ""; ownedClipboard.clear(); } } }
|
|
}
|
|
}
|
|
main().then(receipt => process.stdout.write(JSON.stringify(receipt))).catch(() => { process.stderr.write("Grok private operator callback failed; pairing, OAuth and browser diagnostics withheld"); process.exitCode = 1; });
|