Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/identity-auth.ts
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

155 lines
15 KiB
TypeScript

import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import type { CompanyMemberRecord, CompanyMembersResponse, UserCompanyAccessResponse } from "../../../packages/shared/src/types/access.js";
import { grantsForHumanRole } from "../../../server/src/services/company-member-roles.js";
import { mkdtemp, writeFile, readFile, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { requireSafeInvitationEnrollment } from "./identity-invite-prerequisite.js";
import { privateIdentityBrowser } from "./identity-private-browser.js";
import { randomBytes, randomUUID } from "node:crypto";
import { expect, type Browser, type BrowserContext, type Page } from "@playwright/test";
import type { RunnerApi } from "../api.js";
/** Finite session transfer. Only the private file path reaches Playwright Test. */
export async function installIdentitySession(target: BrowserContext, origin: string, cookie: string) {
const directory = await mkdtemp(path.join(tmpdir(), "identity-human-session-"));
try {
const file = path.join(directory, "storage.json");
await writeFile(file, JSON.stringify({ cookies: cookie.split("; ").filter(Boolean).map(value => ({ name: value.slice(0, value.indexOf("=")), value: value.slice(value.indexOf("=") + 1),
domain: new URL(origin).hostname, path: "/", expires: -1, httpOnly: true, secure: false, sameSite: "Lax" as const })), origins: [] }), { mode: 0o600 });
await target.setStorageState(file);
} finally { await rm(directory, { recursive: true, force: true }); }
}
/** Credentials remain in this closure and the attempt redactor, never case state/evidence.
* Browser installation takes only a private state filename; password entry runs
* in a standalone child with no Playwright Test API-step recorder. */
export async function createAuthenticatedPerson(input: {
api: RunnerApi; browser: Browser; secrets: string[]; name: string; email?: string;
}) {
const health = await input.api.get<{ deploymentMode: string }>("/api/health");
if (health.deploymentMode !== "authenticated") throw new Error("Authenticated-person fixtures require authenticated deployment mode; implicit local board is not a human identity");
const email = input.email ?? `chat-eval-${randomUUID()}@example.test`;
const password = randomBytes(32).toString("base64url"); input.secrets.push(password);
let cookie = "";
async function auth(path: string, data: unknown) {
const response = await fetch(new URL(`/api/auth/${path}`, input.api.baseURL), {
method: "POST", redirect: "error", signal: AbortSignal.timeout(15_000),
headers: { "content-type": "application/json", Origin: input.api.baseURL, ...(cookie ? { Cookie: cookie } : {}) }, body: JSON.stringify(data),
});
if (!response.ok) { await response.body?.cancel(); throw new Error(`Public authentication returned HTTP ${response.status}; details withheld`); }
const cookies = response.headers.getSetCookie().map(value => value.split(";")[0]!).filter(value => value.includes("="));
if (cookies.length) { cookie = cookies.join("; "); for (const value of cookies) input.secrets.push(value.slice(value.indexOf("=") + 1)); }
return response;
}
const response = await auth("sign-up/email", { name: input.name, email, password });
const user = (await response.json() as { user: { id: string; name: string } }).user;
if (!user?.id || !cookie) throw new Error("Signup did not establish a real user session");
const context = await input.browser.newContext({ baseURL: input.api.baseURL });
async function installSession(target: BrowserContext) { await installIdentitySession(target, input.api.baseURL, cookie); }
await installSession(context);
async function request(path: string, method = "GET", data?: unknown) {
const url = new URL(path, input.api.baseURL);
if (url.origin !== input.api.baseURL || !url.pathname.startsWith("/api/")) throw new Error("Person request escaped fixture API");
return fetch(url, { method, redirect: "error", signal: AbortSignal.timeout(15_000), headers: { Cookie: cookie, Origin: input.api.baseURL,
...(data === undefined ? {} : { "content-type": "application/json" }) }, ...(data === undefined ? {} : { body: JSON.stringify(data) }) });
}
return { id: user.id, name: user.name, context, request,
async installSessionOn(target: BrowserContext) { await installSession(target); },
bindApiSession(api: RunnerApi) {
if (api.baseURL !== input.api.baseURL || !cookie) throw new Error("Public human API binding requires the same authenticated instance");
api.setBrowserSession(cookie);
},
/** Server revocation leaves the old browser cookie in place so the next
* navigation exercises an actually invalid session rather than a fake UI flag. */
async expireSession() { await auth("sign-out", {}); cookie = ""; },
async signInThroughBrowser(page: Page, returnPath: string) {
if (page.context() !== context || !returnPath.startsWith("/") || returnPath.startsWith("//")) throw new Error("Human sign-in target must be the person's same-origin browser");
await privateIdentityBrowser({ kind: "sign-in", origin: input.api.baseURL, startPath: page.url(), returnPath, email, password, cookies: [] }, {
async storageState(file) {
const state = JSON.parse(await readFile(file, "utf8")) as { cookies: Array<{ name: string; value: string }> };
cookie = state.cookies.map(value => `${value.name}=${value.value}`).join("; ");
for (const value of state.cookies) input.secrets.push(value.value);
await context.setStorageState(file);
},
});
await page.goto(new URL(returnPath, input.api.baseURL).href, { waitUntil: "domcontentloaded" });
await expect(page).toHaveURL(new URL(returnPath, input.api.baseURL).href);
if (!cookie) throw new Error("Human browser sign-in returned no session");
},
async signOut() { await auth("sign-out", {}); cookie = ""; await context.clearCookies(); },
async signIn() { await auth("sign-in/email", { email, password }); await installSession(context); },
async dispose() { try { if (cookie) await auth("sign-out", {}); } finally { cookie = ""; await context.close(); } },
};
}
/** A member is admitted only by the normal authorized invite and acceptance.
* No synthetic headers, direct memberships, or first-admin claim are used. */
export async function enrollAuthenticatedPerson(api: RunnerApi, companyId: string, person: Awaited<ReturnType<typeof createAuthenticatedPerson>>, secrets: string[], role: "operator" | "viewer" = "operator") {
const invite = await api.postSensitive<{ id: string; token: string }>(`/api/companies/${companyId}/invites`, { allowedJoinTypes: "human", humanRole: role });
secrets.push(invite.token);
await requireSafeInvitationEnrollment(api, companyId, invite.id, role, () => person.dispose());
const response = await person.request(`/api/invites/${encodeURIComponent(invite.token)}/accept`, "POST", { requestType: "human" });
if (!response.ok) { await response.body?.cancel(); throw new Error(`Public join request returned HTTP ${response.status}; details withheld`); }
await response.body?.cancel();
const requests = await api.get<Array<{ id: string; requestingUserId: string; status: string; inviteId: string }>>(`/api/companies/${companyId}/join-requests`);
const pending = requests.filter(row => row.requestingUserId === person.id && row.inviteId === invite.id);
if (pending.length !== 1 || !["pending_approval", "approved"].includes(pending[0]!.status)) throw new Error("Public invitation did not create one eligible scoped join request");
// Human invitation acceptance normally auto-approves through the production
// invited role. Retain support for a deployment's explicit approval gate.
if (pending[0]!.status === "pending_approval") await api.post(`/api/companies/${companyId}/join-requests/${pending[0]!.id}/approve`);
const companiesResponse = await person.request("/api/companies");
if (!companiesResponse.ok || !(await companiesResponse.json() as Array<{ id: string }>).some(row => row.id === companyId)) throw new Error("Approved person cannot read the intended company");
return { userId: person.id, companyId, inviteId: invite.id, joinRequestId: pending[0]!.id };
}
export const prerequisiteMembershipScenarioIds = Object.freeze([
"activity.agent-feedback", "activity.work-timeline", "agents.choose-primary",
"assistant-connections.consent-read-only", "connection-setup.wrong-person-resolves",
"documents-artifacts.private-download", "documents-artifacts.restore-document",
"inbox-search.archive-done", "inbox-search.notification-audience", "inbox-search.restore-archive", "inbox-search.show-decisions",
"navigation-preferences.broken-link", "navigation-preferences.dismiss-announcement", "navigation-preferences.edit-profile", "navigation-preferences.quick-search", "navigation-preferences.sidebar-order",
"projects.private-project", "projects.share-project", "questions-and-approvals.wrong-reviewer", "secrets.personal-secret",
"tasks.completion-with-gate", "tasks.private-task", "tasks.share-private-task",
]);
export async function enrollPrerequisitePerson(ctx: ChatCatalogContext, person: Awaited<ReturnType<typeof createAuthenticatedPerson>>, role: "operator" | "viewer" = "operator") {
if (!prerequisiteMembershipScenarioIds.includes(ctx.scenario.id) || ctx.input.execution.id !== `chat-features.runner-opencode.local.${ctx.scenario.id}` || !["operator", "viewer"].includes(role))
throw new Error("Public prerequisite membership is not admitted for this scenario or role");
const deadline = ctx.observationDeadlineAt;
const remaining = () => { const now = Date.now(); if (!Number.isFinite(deadline) || deadline! <= now) throw new Error("Original prerequisite observation budget exhausted or missing"); return Math.min(15000, deadline! - now); };
const { api, fixtures } = ctx.input, companyId = fixtures.company.id;
const read = async <T>(route: string) => { const value = await api.get<T>(route, { timeout: remaining() }); remaining(); return value; };
const personal = async (route: string) => { remaining(); const response = await person.request(route); remaining(); return response; };
const health = await read<{ deploymentMode: string }>("/api/health");
if (health.deploymentMode !== "authenticated") throw new Error("Public membership requires real authenticated sessions");
const session = await read<{ user: { id: string } }>("/api/auth/get-session");
if (!session.user?.id || session.user.id === person.id || session.user.id === "local-board") throw new Error("Distinct real first-admin authority is required");
const admin = await read<UserCompanyAccessResponse>(`/api/admin/users/${session.user.id}/company-access`);
if (admin.user?.id !== session.user.id || admin.user.isInstanceAdmin !== true) throw new Error("Authenticated fixture administrator authority was not verified");
const targetBefore = await read<UserCompanyAccessResponse>(`/api/admin/users/${person.id}/company-access`);
if (targetBefore.user?.id !== person.id || targetBefore.user.isInstanceAdmin !== false || targetBefore.companyAccess.length !== 0) throw new Error("Prerequisite person must be a fresh non-admin user");
const own = await personal("/api/auth/get-session");
if (own.status !== 200) { await own.body?.cancel(); throw new Error("Prerequisite person's real session unavailable"); }
if (((await own.json()) as { user?: { id?: string } }).user?.id !== person.id) throw new Error("Prerequisite session belongs to another person");
const before = await personal(`/api/companies/${companyId}`), beforeStatus = before.status; await before.body?.cancel();
if (![403, 404].includes(beforeStatus)) throw new Error("Fresh person already has company access or denial proof failed");
remaining(); const assigned = await api.putSensitive<UserCompanyAccessResponse>(`/api/admin/users/${person.id}/company-access`, { companyIds: [companyId] }); remaining();
const membership = assigned.companyAccess.find(row => row.companyId === companyId);
if (assigned.user?.id !== person.id || assigned.user.isInstanceAdmin !== false || assigned.companyAccess.length !== 1 || membership?.principalId !== person.id || membership.principalType !== "user" || membership.status !== "active" || membership.membershipRole !== "operator") throw new Error("Public company access response did not establish the exact prerequisite member");
const grants = grantsForHumanRole(role);
remaining(); await api.patchSensitive<CompanyMemberRecord>(`/api/companies/${companyId}/members/${membership.id}/role-and-grants`, { membershipRole: role, status: "active", grants }); remaining();
const members = await read<CompanyMembersResponse>(`/api/companies/${companyId}/members`), actual = members.members.find(row => row.id === membership.id);
const keys = (rows: Array<{ permissionKey: string; scope?: Record<string, unknown> | null }>) => rows.map(row => `${row.permissionKey}:${JSON.stringify(row.scope ?? null)}`).sort().join("\n");
if (!actual || actual.companyId !== companyId || actual.principalId !== person.id || actual.principalType !== "user" || actual.membershipRole !== role || actual.status !== "active" || keys(actual.grants) !== keys(grants) || actual.grants.some(row => row.companyId !== companyId || row.principalId !== person.id || row.principalType !== "user" || row.grantedByUserId !== session.user.id)) throw new Error("Persisted prerequisite role or grants differ from the public requested contract");
const targetAfter = await read<UserCompanyAccessResponse>(`/api/admin/users/${person.id}/company-access`);
if (targetAfter.user?.id !== person.id || targetAfter.user.isInstanceAdmin !== false || targetAfter.companyAccess.length !== 1 || targetAfter.companyAccess[0]?.id !== membership.id || targetAfter.companyAccess[0]?.companyId !== companyId || targetAfter.companyAccess[0]?.membershipRole !== role || targetAfter.companyAccess[0]?.status !== "active") throw new Error("Prerequisite membership or non-admin identity changed after role assignment");
const allowed = await personal(`/api/companies/${companyId}`);
if (allowed.status !== 200) { await allowed.body?.cancel(); throw new Error("Real prerequisite person cannot read its company"); }
if (((await allowed.json()) as { id?: string }).id !== companyId) throw new Error("Real prerequisite person read another company");
const denied = await personal("/api/admin/users"), deniedStatus = denied.status; await denied.body?.cancel();
if (deniedStatus !== 403) throw new Error("Prerequisite person acquired instance authority or denial proof failed");
const receipt = { userId: person.id, companyId, membershipId: membership.id, role, grantKeys: grants.map(row => row.permissionKey).sort(), authentication: "real-person-session", administrator: "verified-instance-admin", enrollment: "public-company-access", beforeStatus, companyReadStatus: 200, instanceAdminReadStatus: deniedStatus };
remaining(); await ctx.input.evidence("identity-public-prerequisite-membership.json", receipt); remaining();
return receipt;
}