Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/identity-wizard.ts
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

165 lines
16 KiB
TypeScript

import { expect, test, type Page, type Response } from "@playwright/test";
import { privateIdentityBrowser } from "./identity-private-browser.js";
import { pollUntil } from "../api.js";
import type { AiConnectionBinding, AiConnectionList } from "../../../packages/shared/src/ai-connections.js";
export type WizardHarness = "codex_local" | "claude_local";
interface Company { id: string; name: string; issuePrefix: string; defaultResponsibleUserId?: string | null }
interface Agent { id: string; companyId: string; name: string; adapterType: string; adapterConfig: { provider?: string; acpxAgent?: string }; runtimeConfig: { aiConnection?: AiConnectionBinding }; lifecycleState: string; status: string; pauseReason?: string | null }
export interface WizardReceipt {
companyId: string; firstAgentId: string; companyName: string; agentName: string; requestedHarness: WizardHarness;
actualHarness: WizardHarness; environmentId: string; environmentDriver: string; actualProbeEnvironmentId: string;
beforeCompanyIds: string[]; afterCompanyIds: string[]; beforeReadyAgents: number; afterAgentIds: string[];
hello: { httpStatus: number; routeHarness: string; adapterType: string; status: string; checks: Array<{ code: string; level: string }> };
hiredThroughWizard: boolean; firstAgentReady: boolean; chatComposerVisible: boolean; heartbeatRunIds: string[];
readyConversationPath: string;
account: { id: string; grantId: string; provider: string; method: string; status: string; ownership: string; ownerUserId?: string; currentUserId: string; responsibleUserId?: string | null; bindingMode: string };
draftCompanyId?: string; draftAgentId?: string; restoredCompanyId?: string; restoredAgentId?: string; secureFieldRequestedAgain?: boolean;
draftRetainedHarness?: boolean; draftContainsCredential: boolean; publicAgentContainsCredential: boolean;
verificationKind: "public-native-hello-probe"; firstWorkExecuted: false; probeUsage: "unavailable-outside-heartbeat-accounting";
}
/** The actual public wizard owns all company/account/first-agent mutations.
* APIs only read its resulting identities; secrets enter the normal masked
* human field and are never included in receipts or browser draft snapshots. */
interface WizardInput {
page: Page; origin: string; companyName: string; agentName: string; harness: WizardHarness; credential: string;
deadlineAt: number; resume?: boolean; entry?: "empty" | "new-company";
onCompany(company: Company): void; evidence(name: string, value: unknown): Promise<void>;
}
export async function finishIdentityWizard(input: WizardInput): Promise<WizardReceipt> {
const receipt = await privateIdentityBrowser({ kind: "wizard", origin: input.origin, startPath: input.page.url() === "about:blank" ? "/" : input.page.url(), cookies: await input.page.context().cookies(input.origin),
credential: input.credential, companyName: input.companyName, agentName: input.agentName, harness: input.harness, deadlineAt: input.deadlineAt, resume: input.resume, entry: input.entry },
{ company(id) { input.onCompany({ id, name: input.companyName, issuePrefix: "" }); } });
if (!receipt.wizard) throw new Error("Private wizard returned no bounded result");
await input.page.goto(new URL(receipt.wizard.readyConversationPath, input.origin).href, { waitUntil: "domcontentloaded" });
await input.evidence("identity-public-wizard-result.json", receipt.wizard); return receipt.wizard;
}
/** Called only by the standalone worker, never in the Playwright Test process. */
export async function runIdentityWizardInPrivateBrowser(input: WizardInput & { registerCompany(company: Company): Promise<void> }): Promise<WizardReceipt> {
let attached = false; try { test.info(); attached = true; } catch { /* outside Test */ }
if (attached) throw new Error("Credential-bearing wizard must run only in the private child");
const { page, origin } = input; const remaining = () => Math.max(1, Math.min(180_000, input.deadlineAt - Date.now()));
const get = async <T>(endpoint: string): Promise<T> => {
const response = await page.context().request.get(origin + endpoint, { timeout: Math.min(30_000, remaining()) });
if (!response.ok()) throw new Error(`Public wizard read returned HTTP ${response.status()}`); return response.json() as Promise<T>;
};
const before = await get<Company[]>("/api/companies");
if (input.entry === "new-company") {
await page.getByRole("button", { name: /^Open .+ (?:organization|workspace|company) switcher$/i }).click();
await page.getByRole("menuitem", { name: "Create organization", exact: true }).click();
} else await page.goto(`${origin}/onboarding`, { waitUntil: "domcontentloaded" });
await expect(page.getByRole("heading", { name: "What is the name of your organization?", exact: true })).toBeVisible();
await page.locator("#onboarding-company-name").fill(input.companyName);
await page.getByRole("button", { name: "Continue", exact: true }).click();
await expect(page.locator("#onboarding-agent-name")).toBeVisible();
const company = await pollUntil({ label: "wizard-created exact company", deadlineAt: input.deadlineAt,
load: () => get<Company[]>("/api/companies"), accept: rows => rows.filter(row => !before.some(old => old.id === row.id) && row.name === input.companyName).length === 1 }).then(rows => rows.find(row => !before.some(old => old.id === row.id) && row.name === input.companyName)!);
await input.registerCompany(company);
input.onCompany(company);
await page.locator("#onboarding-agent-name").fill(input.agentName); await page.getByRole("button", { name: "Next", exact: true }).click();
await expect(page.getByRole("heading", { name: "Connect a model", exact: true })).toBeVisible();
const setupAgents = await pollUntil({ label: "public first-agent setup record", deadlineAt: input.deadlineAt,
load: () => get<Agent[]>(`/api/companies/${company.id}/agents`), accept: rows => rows.length === 1 && rows[0]?.name === input.agentName });
const provisional = setupAgents[0]!;
const beforeReadyAgents = setupAgents.filter(row => row.lifecycleState === "ready" && ["idle", "active"].includes(row.status)).length;
if (beforeReadyAgents) throw new Error("Wizard prerequisite already had a usable first agent before account setup");
const environments = await get<Array<{ id: string; driver: string }>>(`/api/companies/${company.id}/environments?driver=local`);
if (environments.length !== 1 || environments[0]?.driver !== "local") throw new Error("Wizard requires one real available local target; no silent target substitution is allowed");
const environment = environments[0]!;
const choose = async () => {
await page.getByRole("radio", { name: input.harness === "claude_local" ? /^Claude/ : /^OpenAI/ }).click();
const apiMode = page.getByRole("button", { name: "Use API key instead", exact: true });
const key = page.getByPlaceholder("Enter API key here", { exact: true });
await expect(apiMode.or(key).first()).toBeVisible();
if (await apiMode.isVisible()) await apiMode.click();
await expect(key).toBeVisible(); return key;
};
let key = await choose();
let draftCompanyId: string | undefined, draftAgentId: string | undefined, restoredCompanyId: string | undefined, restoredAgentId: string | undefined;
let secureFieldRequestedAgain: boolean | undefined, draftRetainedHarness: boolean | undefined;
await key.fill(input.credential);
// Retain only booleans/identity fields from browser-local storage. Never emit
// its raw JSON, which is exactly the privacy surface under test.
const draft = async () => page.evaluate(secret => {
const raw = localStorage.getItem("paperclip-onboarding-state") ?? ""; let state: Record<string, unknown> = {};
try { state = JSON.parse(raw) as Record<string, unknown>; } catch { /* Readiness below stays false. */ }
return { companyId: typeof state.createdCompanyId === "string" ? state.createdCompanyId : null,
agentId: typeof state.setupAgentId === "string" ? state.setupAgentId : null, harness: typeof state.adapterType === "string" ? state.adapterType : null,
containsCredential: Object.values(localStorage).some(value => String(value).includes(secret)) };
}, input.credential);
const saved = await pollUntil({ label: "nonsecret wizard draft persisted", deadlineAt: input.deadlineAt, load: draft,
accept: value => value.companyId === company.id && value.agentId === provisional.id && value.harness === input.harness });
if (saved.containsCredential) throw new Error("The actual wizard draft leaked its secure input; contents withheld");
if (input.resume) {
draftCompanyId = saved.companyId!; draftAgentId = saved.agentId!;
await page.reload({ waitUntil: "domcontentloaded" });
await expect(page.getByRole("heading", { name: "Connect a model", exact: true })).toBeVisible();
const restored = await draft(); restoredCompanyId = restored.companyId!; restoredAgentId = restored.agentId!; draftRetainedHarness = restored.harness === input.harness;
key = page.getByPlaceholder("Enter API key here", { exact: true }); await expect(key).toBeVisible();
secureFieldRequestedAgain = await key.inputValue() === "";
if (!secureFieldRequestedAgain || restored.containsCredential) throw new Error("Reload did not request the secure input again; values withheld");
const existing = await get<Agent[]>(`/api/companies/${company.id}/agents`);
if (existing.length !== 1 || existing[0]?.id !== provisional.id || existing[0]?.lifecycleState === "ready") throw new Error("Wizard reload duplicated or prematurely completed the first agent");
await key.fill(input.credential);
}
let hello: WizardReceipt["hello"] | undefined; let actualProbeEnvironmentId = ""; let hiredThroughWizard = false;
const reads = new Set<Promise<void>>();
const onResponse = (response: Response) => {
const url = new URL(response.url()); if (url.origin !== origin || response.request().method() !== "POST") return;
const test = url.pathname === `/api/companies/${company.id}/adapters/${input.harness}/test-environment`;
const hire = url.pathname === `/api/companies/${company.id}/agent-hires`;
if (!test && !hire) return;
const pending = (async () => {
if (hire) { hiredThroughWizard = response.ok(); return; }
const request = response.request().postDataJSON() as { environmentId?: unknown };
actualProbeEnvironmentId = typeof request.environmentId === "string" ? request.environmentId : "";
const body = await response.json() as { adapterType?: string; status?: string; checks?: Array<{ code?: string; level?: string }> };
hello = { httpStatus: response.status(), routeHarness: input.harness, adapterType: body.adapterType ?? "", status: body.status ?? "",
checks: (body.checks ?? []).map(row => ({ code: row.code ?? "", level: row.level ?? "" })) };
})(); reads.add(pending); void pending.finally(() => reads.delete(pending)).catch(() => {});
};
page.on("response", onResponse);
try {
await page.getByRole("button", { name: /^(?:Connect|Next)$/, exact: true }).last().click();
await expect(page.getByRole("button", { name: "Get started", exact: true })).toBeVisible({ timeout: remaining() });
await Promise.all([...reads]);
} finally { page.off("response", onResponse); }
const expectedHello = input.harness === "claude_local" ? ["claude_hello_probe_passed", "acpx_hello_probe_passed"] : ["codex_hello_probe_passed"];
if (!hello || hello.httpStatus !== 200 || !["pass", "warn"].includes(hello.status) || hello.checks.some(row => row.level === "error" || row.code === "adapter_auth_missing" || /_auth_required$/.test(row.code)) || !hello.checks.some(row => expectedHello.includes(row.code))
|| actualProbeEnvironmentId !== environment.id || !hiredThroughWizard) throw new Error("Actual wizard account/harness/target verification is incomplete; a saved profile alone is not readiness");
const agents = await get<Agent[]>(`/api/companies/${company.id}/agents`);
if (agents.length !== 1 || agents[0]?.id !== provisional.id) throw new Error("Wizard completion duplicated or replaced the saved first agent");
const agent = await pollUntil({ label: "wizard first agent independently ready", deadlineAt: input.deadlineAt,
load: () => get<Agent>(`/api/agents/${provisional.id}`), accept: row => row.lifecycleState === "ready" && ["idle", "active"].includes(row.status) });
const actualHarness = agent.adapterType === "claude_local" || agent.adapterType === "codex_local" ? agent.adapterType
: agent.adapterType === "paperclip_runner" && agent.adapterConfig.provider === "acpx" && agent.adapterConfig.acpxAgent === "claude" ? "claude_local"
: agent.adapterType === "paperclip_runner" && agent.adapterConfig.provider === "codex" ? "codex_local" : undefined;
if (actualHarness !== input.harness || agent.companyId !== company.id || JSON.stringify(agent).includes(input.credential)) throw new Error("Public ready agent differs from the requested harness/company or exposes its credential; contents withheld");
const accounts = await get<AiConnectionList>(`/api/companies/${company.id}/ai-connections`);
const responsibleCompany = await get<Company>(`/api/companies/${company.id}`);
const provider = input.harness === "claude_local" ? "anthropic" : "openai";
const account = accounts.connections.find(row => row.companyId === company.id && row.provider === provider && row.method === "api_key" && row.ownership === "personal"
&& row.ownerUserId === accounts.currentUserId && row.isDefault && row.status === "connected");
if (!account || responsibleCompany.defaultResponsibleUserId !== accounts.currentUserId || agent.runtimeConfig.aiConnection?.provider !== provider
|| agent.runtimeConfig.aiConnection.method !== "api_key" || agent.runtimeConfig.aiConnection.mode !== "responsible_user") throw new Error("Ready wizard agent does not retain the actual responsible person's selected provider account");
await page.getByRole("button", { name: "Get started", exact: true }).click();
await expect(page.getByTestId("task-chat-composer-input").or(page.getByTestId("task-chat-composer-takeover")).first()).toBeVisible();
const settings = await get<{ enableAgentChat: boolean }>("/api/instance/settings/experimental");
if (settings.enableAgentChat) {
await page.goto(`${origin}/${company.issuePrefix}/chats/${agent.id}`, { waitUntil: "domcontentloaded" });
await expect(page.getByTestId("task-chat-composer-input")).toBeVisible();
}
const runIds = (await get<Array<{ id: string }>>(`/api/companies/${company.id}/heartbeat-runs?limit=100`)).map(row => row.id);
const afterCompanies = await get<Company[]>("/api/companies");
const finalDraft = await draft();
const receipt: WizardReceipt = { companyId: company.id, firstAgentId: agent.id, companyName: company.name, agentName: agent.name, requestedHarness: input.harness, actualHarness,
environmentId: environment.id, environmentDriver: environment.driver, actualProbeEnvironmentId, beforeCompanyIds: before.map(row => row.id), afterCompanyIds: afterCompanies.map(row => row.id),
beforeReadyAgents, afterAgentIds: agents.map(row => row.id), hello, hiredThroughWizard, firstAgentReady: true, chatComposerVisible: true, heartbeatRunIds: runIds, readyConversationPath: new URL(page.url()).pathname,
account: { id: account.id, grantId: account.grantId, provider: account.provider, method: account.method, status: account.status, ownership: account.ownership,
ownerUserId: account.ownerUserId, currentUserId: accounts.currentUserId, responsibleUserId: responsibleCompany.defaultResponsibleUserId, bindingMode: agent.runtimeConfig.aiConnection.mode },
...(input.resume ? { draftCompanyId, draftAgentId, restoredCompanyId, restoredAgentId, secureFieldRequestedAgain, draftRetainedHarness } : {}),
draftContainsCredential: saved.containsCredential || finalDraft.containsCredential, publicAgentContainsCredential: false,
verificationKind: "public-native-hello-probe", firstWorkExecuted: false, probeUsage: "unavailable-outside-heartbeat-accounting" };
await input.evidence("identity-public-wizard-result.json", receipt); return receipt;
}