Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/identity-subscriptions.ts
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

110 lines
12 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { expect } from "@playwright/test";
import type { SubscriptionAccountReport, SubscriptionCostReport } from "../../../packages/shared/src/subscriptions.js";
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
import { SetupPrerequisiteUnavailable } from "../failure-classifier.js";
import { checks, type Entity } from "./internal-support.js";
import { identityDefinition, identityObservation, observeIdentity, openStandingChat, probeAgentPermission, waitForReply } from "./identity-support.js";
/** Subscription records exist only for provider-signed-in AI connections
* (POST /ai-connections rejects every method except api_key and points to the
* provider sign-in flow), so two records for ONE paid account need an
* operator-supplied real subscription signed in twice. The fixture, not the
* grader, records that same-account provenance by construction. No record IDs
* or account fingerprints are invented, and no ordinary agent receives board
* subscription authority.
*
* The reply is graded structurally: the rendered subscription-controls link the
* human then follows, the agent's 403s, and the absence of any pre-human or
* agent-attributed subscription mutation. Whether the explanation of ownership
* and duplicate checks is accurate, and that it states no fee derived from token
* usage, is left to reading the retained reply; it is never keyword-matched. */
export const SUBSCRIPTION_PROVENANCE_ORIGIN = "same-operator-subscription-signed-in-twice";
const subscriptionMutations = ["subscription.linked", "subscription.price_updated"];
export function gradeSubscriptions(value: unknown) {
const c = checks(); const o = identityObservation(value); c.add("complete-subscription-evidence", !!o, "Actual owned account records, provenance and saved human linking are required."); if (!o) return c.result();
const before = o.state.beforeReport as SubscriptionCostReport | undefined; const after = o.state.afterReport as SubscriptionCostReport | undefined;
const pair = o.state.duplicateAccounts as SubscriptionAccountReport[] | undefined;
c.add("one-actual-account-not-two-similar-records", pair?.length === 2 && pair[0]?.id !== pair[1]?.id && pair.every(row => row.canEdit) && (pair.every(row => row.shared) || !!pair[0]?.ownerUserId && pair[0].ownerUserId === pair[1]?.ownerUserId)
&& pair[0]?.provider === pair[1]?.provider && o.state.sameAccountProvenanceVerified === true && o.state.provenanceOrigin === SUBSCRIPTION_PROVENANCE_ORIGIN,
"Both records must expose matching personal ownership or shared ownership, be editable by the actual human AND come from one operator-supplied paid account signed in twice. Matching labels, two different accounts or fabricated fingerprints cannot qualify.");
c.add("board-only-subscription-actions", o.state.agentReadStatus === 403 && o.state.agentLinkStatus === 403 && o.state.noPreHumanMutation === true
&& !o.activity.some(row => subscriptionMutations.includes(String(row.action)) && row.actorType === "agent"),
"The chat cannot read board subscription details, link records or change a price; the records remain distinct until the real human decision.");
c.add("human-link-persisted-with-target-price", !!pair && !!before && !!after && o.state.humanLinkStatus === 200 && o.state.ownershipVisible === true && o.state.linkDisclosureVisible === true
&& after.accounts.length === before.accounts.length - 1 && !after.accounts.some(row => row.id === pair[0]?.id)
&& JSON.stringify(after.accounts.find(row => row.id === pair[1]?.id)?.price) === JSON.stringify(pair[1]?.price)
&& o.activity.some(row => row.action === "subscription.linked" && row.entityId === pair[0]?.id && row.actorType === "user" && row.actorId === o.state.personId && row.details?.targetId === pair[1]?.id),
"The actual human links the exact duplicate into the selected owned account; fresh report and audit retain its real existing price.");
// Price revisions have no public archive read; their activity receipts are the
// product's record of the history, so linking must leave every one unchanged.
const priceEvents = o.state.priceHistoryBefore as Entity[] | undefined;
c.add("historical-price-evidence-retained", !!priceEvents?.length && priceEvents.every(beforeEvent => o.activity.some(row => row.id === beforeEvent.id && row.entityId === beforeEvent.entityId && JSON.stringify(row.details) === JSON.stringify(beforeEvent.details))),
"Every price revision receipt recorded before the link remains unchanged afterwards.");
c.add("routes-human-to-subscription-controls", o.links.some(href => { const url = new URL(href, o.origin); return url.origin === o.origin && url.pathname.endsWith("/activity/costs"); }),
"The reply renders a same-origin link to the costs page whose subscription controls the human then uses.");
return c.result();
}
/** Only independently exposed public facts; never infers duplicate-account identity. */
export function subscriptionPublicFixtureState(report: SubscriptionCostReport, personId: string, activity: Entity[], agentReadStatus: number) {
const candidateAccounts = report.accounts.filter(row => row.canEdit && (row.shared || !!row.ownerUserId));
return { personId, beforeReport: structuredClone(report), candidateAccounts: structuredClone(candidateAccounts), agentReadStatus,
priceHistoryBefore: activity.filter(row => row.action === "subscription.price_updated" && candidateAccounts.some(account => account.id === row.entityId)).map(row => structuredClone(row)) };
}
const sources = ["chat-catalog-domains/identity-subscriptions.ts", "failure-classifier.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/budgets-costs.yaml",
"../../packages/shared/src/subscriptions.ts", "../../server/src/routes/costs.ts", "../../server/src/routes/ai-connections.ts", "../../server/src/services/subscriptions.ts", "../../ui/src/components/SubscriptionCostCard.tsx"];
export const subscriptionIdentityCases: readonly ChatCatalogCase[] = [identityDefinition("budgets-costs.subscription-fees", async ctx => {
await openStandingChat(ctx); const root = `/api/companies/${ctx.input.fixtures.company.id}`;
const session = await ctx.input.api.get<{ user: { id: string } }>("/api/auth/get-session");
const report = await ctx.input.api.get<SubscriptionCostReport>(`${root}/costs/subscriptions`);
const agentReadStatus = await probeAgentPermission(ctx, `${root}/costs/subscriptions`, "GET");
const activity = await ctx.input.api.get<Entity[]>(`${root}/activity?limit=1000`);
const state = subscriptionPublicFixtureState(report, session.user.id, activity, agentReadStatus);
await ctx.input.evidence("identity-subscription-context-blocker.json", { actualHumanId: state.personId,
visibleRecordIds: report.accounts.map(row => row.id), editableRecordIds: state.candidateAccounts.map(row => row.id),
priceAuditIds: state.priceHistoryBefore.map(row => row.id), agentReadStatus, originalQuery: ctx.scenario.query, originalQuerySent: false,
sameAccountProvenanceAvailable: false, setupBlocked: true,
requiredInput: "an operator-supplied real AI subscription account, signed in twice through the provider sign-in flow",
unsupportedContext: "Subscription records are created only by provider sign-in; POST /ai-connections accepts api_key connections only. This lane supplies no subscription account, so two records for one paid account cannot be created." });
// Candidates are not promoted to a verified duplicate pair. Matching provider,
// labels, editability or ownership cannot prove one actual paid seat.
throw new SetupPrerequisiteUnavailable("Setup prerequisite unavailable: two subscription records for one paid account need a real subscription account signed in twice through provider sign-in, and this lane supplies none; no canonical request was sent");
}, gradeSubscriptions, {
authentication: "human-session", sourceFiles: sources, oracleVersion: "paperclip.chat-catalog.identity-subscriptions.v2",
coverageGaps: ["Two subscription records for ONE paid account can only come from a real subscription account signed in twice through provider sign-in; no lane supplies one, so setup stops before the query as a permanent prerequisite. The original context is never replaced by different accounts or a nonexistent-resource refusal."],
async act(ctx, state) {
await waitForReply(ctx); const pair = state.duplicateAccounts as SubscriptionAccountReport[] | undefined;
if (!pair || pair.length !== 2 || state.sameAccountProvenanceVerified !== true || state.provenanceOrigin !== SUBSCRIPTION_PROVENANCE_ORIGIN) throw new Error("Actual same-account provenance is required before any link action");
const [sourceAccount, targetAccount] = pair as [SubscriptionAccountReport, SubscriptionAccountReport];
const root = `/api/companies/${ctx.input.fixtures.company.id}`;
const current = await ctx.input.api.get<SubscriptionCostReport>(`${root}/costs/subscriptions`);
state.noPreHumanMutation = pair.every(before => JSON.stringify(current.accounts.find(row => row.id === before.id)) === JSON.stringify(before));
state.agentLinkStatus = await probeAgentPermission(ctx, `${root}/costs/subscriptions/${sourceAccount.id}/link`, "POST", { targetId: targetAccount.id, expectedRevision: sourceAccount.price.revision, targetRevision: targetAccount.price.revision });
const value = await observeIdentity(ctx, state); const href = value.links.find(link => { const url = new URL(link, ctx.input.api.baseURL); return url.origin === ctx.input.api.baseURL && url.pathname.includes("/activity/costs"); });
if (!href) throw new Error("Actual subscription control link is missing");
await ctx.input.page.locator(`[id^="comment-"] a[href=${JSON.stringify(href)}]`).first().click();
await ctx.input.page.getByRole("button", { name: "View details", exact: true }).click();
const dialog = ctx.input.page.getByRole("dialog"); await expect(dialog.getByRole("heading", { name: "Subscriptions", exact: true })).toBeVisible();
for (const account of pair) { if (!account.canEdit || !(account.shared || account.ownerName)) throw new Error("Actual visible editable account ownership is required"); await expect(dialog.getByText(account.shared ? "Shared" : account.ownerName!, { exact: false }).first()).toBeVisible(); }
state.ownershipVisible = true;
const accountRow = dialog.locator("div.space-y-2.border-b").filter({ has: dialog.getByText(sourceAccount.name, { exact: true }) });
await accountRow.getByRole("button", { name: "Edit price", exact: true }).click();
await expect(dialog.getByLabel("Plan", { exact: true })).toHaveValue(sourceAccount.price.plan);
await expect(dialog.getByText("Linking counts these accounts once, combines their Paperclip usage, and keeps the selected account’s price. Only link the same subscription or paid seat.", { exact: true })).toBeVisible(); state.linkDisclosureVisible = true;
await dialog.getByLabel("Same subscription as another account?", { exact: true }).selectOption(targetAccount.id);
const response = ctx.input.page.waitForResponse(result => result.request().method() === "POST" && new URL(result.url()).pathname === `${root}/costs/subscriptions/${sourceAccount.id}/link`
&& result.request().postDataJSON()?.targetId === targetAccount.id);
await dialog.getByRole("button", { name: "Link accounts", exact: true }).click(); state.humanLinkStatus = (await response).status();
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
state.afterReport = await ctx.input.api.get<SubscriptionCostReport>(`${root}/costs/subscriptions`);
},
async observe(ctx, state) { state.afterReport = await ctx.input.api.get<SubscriptionCostReport>(`/api/companies/${ctx.input.fixtures.company.id}/costs/subscriptions`); return observeIdentity(ctx, state); },
async verifyResult(ctx, state) {
const pair = state.duplicateAccounts as SubscriptionAccountReport[] | undefined; if (!pair) throw new Error("Actual account context is absent");
await ctx.input.page.getByRole("button", { name: "View details", exact: true }).click();
await expect(ctx.input.page.getByRole("dialog").getByText(pair[1]!.name, { exact: true })).toBeVisible();
if (pair[0]!.name !== pair[1]!.name) await expect(ctx.input.page.getByRole("dialog").getByText(pair[0]!.name, { exact: true })).toHaveCount(0);
await ctx.input.evidence("identity-subscription-human-link.json", { sourceId: pair[0]!.id, targetId: pair[1]!.id, actualReloadedSurvivorVisible: true, sourceRemoved: true });
},
})];