mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 18:56:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
155 lines
15 KiB
TypeScript
155 lines
15 KiB
TypeScript
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
import type { CompanyMemberRecord, CompanyMembersResponse, UserCompanyAccessResponse } from "../../../packages/shared/src/types/access.js";
|
|
import { grantsForHumanRole } from "../../../server/src/services/company-member-roles.js";
|
|
import { mkdtemp, writeFile, readFile, rm } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import { requireSafeInvitationEnrollment } from "./identity-invite-prerequisite.js";
|
|
import { privateIdentityBrowser } from "./identity-private-browser.js";
|
|
import { randomBytes, randomUUID } from "node:crypto";
|
|
import { expect, type Browser, type BrowserContext, type Page } from "@playwright/test";
|
|
import type { RunnerApi } from "../api.js";
|
|
|
|
/** Finite session transfer. Only the private file path reaches Playwright Test. */
|
|
export async function installIdentitySession(target: BrowserContext, origin: string, cookie: string) {
|
|
const directory = await mkdtemp(path.join(tmpdir(), "identity-human-session-"));
|
|
try {
|
|
const file = path.join(directory, "storage.json");
|
|
await writeFile(file, JSON.stringify({ cookies: cookie.split("; ").filter(Boolean).map(value => ({ name: value.slice(0, value.indexOf("=")), value: value.slice(value.indexOf("=") + 1),
|
|
domain: new URL(origin).hostname, path: "/", expires: -1, httpOnly: true, secure: false, sameSite: "Lax" as const })), origins: [] }), { mode: 0o600 });
|
|
await target.setStorageState(file);
|
|
} finally { await rm(directory, { recursive: true, force: true }); }
|
|
}
|
|
|
|
/** Credentials remain in this closure and the attempt redactor, never case state/evidence.
|
|
* Browser installation takes only a private state filename; password entry runs
|
|
* in a standalone child with no Playwright Test API-step recorder. */
|
|
export async function createAuthenticatedPerson(input: {
|
|
api: RunnerApi; browser: Browser; secrets: string[]; name: string; email?: string;
|
|
}) {
|
|
const health = await input.api.get<{ deploymentMode: string }>("/api/health");
|
|
if (health.deploymentMode !== "authenticated") throw new Error("Authenticated-person fixtures require authenticated deployment mode; implicit local board is not a human identity");
|
|
const email = input.email ?? `chat-eval-${randomUUID()}@example.test`;
|
|
const password = randomBytes(32).toString("base64url"); input.secrets.push(password);
|
|
let cookie = "";
|
|
async function auth(path: string, data: unknown) {
|
|
const response = await fetch(new URL(`/api/auth/${path}`, input.api.baseURL), {
|
|
method: "POST", redirect: "error", signal: AbortSignal.timeout(15_000),
|
|
headers: { "content-type": "application/json", Origin: input.api.baseURL, ...(cookie ? { Cookie: cookie } : {}) }, body: JSON.stringify(data),
|
|
});
|
|
if (!response.ok) { await response.body?.cancel(); throw new Error(`Public authentication returned HTTP ${response.status}; details withheld`); }
|
|
const cookies = response.headers.getSetCookie().map(value => value.split(";")[0]!).filter(value => value.includes("="));
|
|
if (cookies.length) { cookie = cookies.join("; "); for (const value of cookies) input.secrets.push(value.slice(value.indexOf("=") + 1)); }
|
|
return response;
|
|
}
|
|
const response = await auth("sign-up/email", { name: input.name, email, password });
|
|
const user = (await response.json() as { user: { id: string; name: string } }).user;
|
|
if (!user?.id || !cookie) throw new Error("Signup did not establish a real user session");
|
|
const context = await input.browser.newContext({ baseURL: input.api.baseURL });
|
|
async function installSession(target: BrowserContext) { await installIdentitySession(target, input.api.baseURL, cookie); }
|
|
await installSession(context);
|
|
async function request(path: string, method = "GET", data?: unknown) {
|
|
const url = new URL(path, input.api.baseURL);
|
|
if (url.origin !== input.api.baseURL || !url.pathname.startsWith("/api/")) throw new Error("Person request escaped fixture API");
|
|
return fetch(url, { method, redirect: "error", signal: AbortSignal.timeout(15_000), headers: { Cookie: cookie, Origin: input.api.baseURL,
|
|
...(data === undefined ? {} : { "content-type": "application/json" }) }, ...(data === undefined ? {} : { body: JSON.stringify(data) }) });
|
|
}
|
|
return { id: user.id, name: user.name, context, request,
|
|
async installSessionOn(target: BrowserContext) { await installSession(target); },
|
|
bindApiSession(api: RunnerApi) {
|
|
if (api.baseURL !== input.api.baseURL || !cookie) throw new Error("Public human API binding requires the same authenticated instance");
|
|
api.setBrowserSession(cookie);
|
|
},
|
|
/** Server revocation leaves the old browser cookie in place so the next
|
|
* navigation exercises an actually invalid session rather than a fake UI flag. */
|
|
async expireSession() { await auth("sign-out", {}); cookie = ""; },
|
|
async signInThroughBrowser(page: Page, returnPath: string) {
|
|
if (page.context() !== context || !returnPath.startsWith("/") || returnPath.startsWith("//")) throw new Error("Human sign-in target must be the person's same-origin browser");
|
|
await privateIdentityBrowser({ kind: "sign-in", origin: input.api.baseURL, startPath: page.url(), returnPath, email, password, cookies: [] }, {
|
|
async storageState(file) {
|
|
const state = JSON.parse(await readFile(file, "utf8")) as { cookies: Array<{ name: string; value: string }> };
|
|
cookie = state.cookies.map(value => `${value.name}=${value.value}`).join("; ");
|
|
for (const value of state.cookies) input.secrets.push(value.value);
|
|
await context.setStorageState(file);
|
|
},
|
|
});
|
|
await page.goto(new URL(returnPath, input.api.baseURL).href, { waitUntil: "domcontentloaded" });
|
|
await expect(page).toHaveURL(new URL(returnPath, input.api.baseURL).href);
|
|
if (!cookie) throw new Error("Human browser sign-in returned no session");
|
|
},
|
|
async signOut() { await auth("sign-out", {}); cookie = ""; await context.clearCookies(); },
|
|
async signIn() { await auth("sign-in/email", { email, password }); await installSession(context); },
|
|
async dispose() { try { if (cookie) await auth("sign-out", {}); } finally { cookie = ""; await context.close(); } },
|
|
};
|
|
}
|
|
|
|
/** A member is admitted only by the normal authorized invite and acceptance.
|
|
* No synthetic headers, direct memberships, or first-admin claim are used. */
|
|
export async function enrollAuthenticatedPerson(api: RunnerApi, companyId: string, person: Awaited<ReturnType<typeof createAuthenticatedPerson>>, secrets: string[], role: "operator" | "viewer" = "operator") {
|
|
const invite = await api.postSensitive<{ id: string; token: string }>(`/api/companies/${companyId}/invites`, { allowedJoinTypes: "human", humanRole: role });
|
|
secrets.push(invite.token);
|
|
await requireSafeInvitationEnrollment(api, companyId, invite.id, role, () => person.dispose());
|
|
const response = await person.request(`/api/invites/${encodeURIComponent(invite.token)}/accept`, "POST", { requestType: "human" });
|
|
if (!response.ok) { await response.body?.cancel(); throw new Error(`Public join request returned HTTP ${response.status}; details withheld`); }
|
|
await response.body?.cancel();
|
|
const requests = await api.get<Array<{ id: string; requestingUserId: string; status: string; inviteId: string }>>(`/api/companies/${companyId}/join-requests`);
|
|
const pending = requests.filter(row => row.requestingUserId === person.id && row.inviteId === invite.id);
|
|
if (pending.length !== 1 || !["pending_approval", "approved"].includes(pending[0]!.status)) throw new Error("Public invitation did not create one eligible scoped join request");
|
|
// Human invitation acceptance normally auto-approves through the production
|
|
// invited role. Retain support for a deployment's explicit approval gate.
|
|
if (pending[0]!.status === "pending_approval") await api.post(`/api/companies/${companyId}/join-requests/${pending[0]!.id}/approve`);
|
|
const companiesResponse = await person.request("/api/companies");
|
|
if (!companiesResponse.ok || !(await companiesResponse.json() as Array<{ id: string }>).some(row => row.id === companyId)) throw new Error("Approved person cannot read the intended company");
|
|
return { userId: person.id, companyId, inviteId: invite.id, joinRequestId: pending[0]!.id };
|
|
}
|
|
|
|
export const prerequisiteMembershipScenarioIds = Object.freeze([
|
|
"activity.agent-feedback", "activity.work-timeline", "agents.choose-primary",
|
|
"assistant-connections.consent-read-only", "connection-setup.wrong-person-resolves",
|
|
"documents-artifacts.private-download", "documents-artifacts.restore-document",
|
|
"inbox-search.archive-done", "inbox-search.notification-audience", "inbox-search.restore-archive", "inbox-search.show-decisions",
|
|
"navigation-preferences.broken-link", "navigation-preferences.dismiss-announcement", "navigation-preferences.edit-profile", "navigation-preferences.quick-search", "navigation-preferences.sidebar-order",
|
|
"projects.private-project", "projects.share-project", "questions-and-approvals.wrong-reviewer", "secrets.personal-secret",
|
|
"tasks.completion-with-gate", "tasks.private-task", "tasks.share-private-task",
|
|
]);
|
|
export async function enrollPrerequisitePerson(ctx: ChatCatalogContext, person: Awaited<ReturnType<typeof createAuthenticatedPerson>>, role: "operator" | "viewer" = "operator") {
|
|
if (!prerequisiteMembershipScenarioIds.includes(ctx.scenario.id) || ctx.input.execution.id !== `chat-features.runner-opencode.local.${ctx.scenario.id}` || !["operator", "viewer"].includes(role))
|
|
throw new Error("Public prerequisite membership is not admitted for this scenario or role");
|
|
const deadline = ctx.observationDeadlineAt;
|
|
const remaining = () => { const now = Date.now(); if (!Number.isFinite(deadline) || deadline! <= now) throw new Error("Original prerequisite observation budget exhausted or missing"); return Math.min(15000, deadline! - now); };
|
|
const { api, fixtures } = ctx.input, companyId = fixtures.company.id;
|
|
const read = async <T>(route: string) => { const value = await api.get<T>(route, { timeout: remaining() }); remaining(); return value; };
|
|
const personal = async (route: string) => { remaining(); const response = await person.request(route); remaining(); return response; };
|
|
const health = await read<{ deploymentMode: string }>("/api/health");
|
|
if (health.deploymentMode !== "authenticated") throw new Error("Public membership requires real authenticated sessions");
|
|
const session = await read<{ user: { id: string } }>("/api/auth/get-session");
|
|
if (!session.user?.id || session.user.id === person.id || session.user.id === "local-board") throw new Error("Distinct real first-admin authority is required");
|
|
const admin = await read<UserCompanyAccessResponse>(`/api/admin/users/${session.user.id}/company-access`);
|
|
if (admin.user?.id !== session.user.id || admin.user.isInstanceAdmin !== true) throw new Error("Authenticated fixture administrator authority was not verified");
|
|
const targetBefore = await read<UserCompanyAccessResponse>(`/api/admin/users/${person.id}/company-access`);
|
|
if (targetBefore.user?.id !== person.id || targetBefore.user.isInstanceAdmin !== false || targetBefore.companyAccess.length !== 0) throw new Error("Prerequisite person must be a fresh non-admin user");
|
|
const own = await personal("/api/auth/get-session");
|
|
if (own.status !== 200) { await own.body?.cancel(); throw new Error("Prerequisite person's real session unavailable"); }
|
|
if (((await own.json()) as { user?: { id?: string } }).user?.id !== person.id) throw new Error("Prerequisite session belongs to another person");
|
|
const before = await personal(`/api/companies/${companyId}`), beforeStatus = before.status; await before.body?.cancel();
|
|
if (![403, 404].includes(beforeStatus)) throw new Error("Fresh person already has company access or denial proof failed");
|
|
remaining(); const assigned = await api.putSensitive<UserCompanyAccessResponse>(`/api/admin/users/${person.id}/company-access`, { companyIds: [companyId] }); remaining();
|
|
const membership = assigned.companyAccess.find(row => row.companyId === companyId);
|
|
if (assigned.user?.id !== person.id || assigned.user.isInstanceAdmin !== false || assigned.companyAccess.length !== 1 || membership?.principalId !== person.id || membership.principalType !== "user" || membership.status !== "active" || membership.membershipRole !== "operator") throw new Error("Public company access response did not establish the exact prerequisite member");
|
|
const grants = grantsForHumanRole(role);
|
|
remaining(); await api.patchSensitive<CompanyMemberRecord>(`/api/companies/${companyId}/members/${membership.id}/role-and-grants`, { membershipRole: role, status: "active", grants }); remaining();
|
|
const members = await read<CompanyMembersResponse>(`/api/companies/${companyId}/members`), actual = members.members.find(row => row.id === membership.id);
|
|
const keys = (rows: Array<{ permissionKey: string; scope?: Record<string, unknown> | null }>) => rows.map(row => `${row.permissionKey}:${JSON.stringify(row.scope ?? null)}`).sort().join("\n");
|
|
if (!actual || actual.companyId !== companyId || actual.principalId !== person.id || actual.principalType !== "user" || actual.membershipRole !== role || actual.status !== "active" || keys(actual.grants) !== keys(grants) || actual.grants.some(row => row.companyId !== companyId || row.principalId !== person.id || row.principalType !== "user" || row.grantedByUserId !== session.user.id)) throw new Error("Persisted prerequisite role or grants differ from the public requested contract");
|
|
const targetAfter = await read<UserCompanyAccessResponse>(`/api/admin/users/${person.id}/company-access`);
|
|
if (targetAfter.user?.id !== person.id || targetAfter.user.isInstanceAdmin !== false || targetAfter.companyAccess.length !== 1 || targetAfter.companyAccess[0]?.id !== membership.id || targetAfter.companyAccess[0]?.companyId !== companyId || targetAfter.companyAccess[0]?.membershipRole !== role || targetAfter.companyAccess[0]?.status !== "active") throw new Error("Prerequisite membership or non-admin identity changed after role assignment");
|
|
const allowed = await personal(`/api/companies/${companyId}`);
|
|
if (allowed.status !== 200) { await allowed.body?.cancel(); throw new Error("Real prerequisite person cannot read its company"); }
|
|
if (((await allowed.json()) as { id?: string }).id !== companyId) throw new Error("Real prerequisite person read another company");
|
|
const denied = await personal("/api/admin/users"), deniedStatus = denied.status; await denied.body?.cancel();
|
|
if (deniedStatus !== 403) throw new Error("Prerequisite person acquired instance authority or denial proof failed");
|
|
const receipt = { userId: person.id, companyId, membershipId: membership.id, role, grantKeys: grants.map(row => row.permissionKey).sort(), authentication: "real-person-session", administrator: "verified-instance-admin", enrollment: "public-company-access", beforeStatus, companyReadStatus: 200, instanceAdminReadStatus: deniedStatus };
|
|
remaining(); await ctx.input.evidence("identity-public-prerequisite-membership.json", receipt); remaining();
|
|
return receipt;
|
|
}
|