mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
57 lines
7.6 KiB
TypeScript
57 lines
7.6 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import type { SubscriptionAccountReport, SubscriptionCostReport, SubscriptionUsage } from "../../../packages/shared/src/subscriptions.js";
|
|
import type { IdentityObservation } from "./identity-support.js";
|
|
import { SUBSCRIPTION_PROVENANCE_ORIGIN, subscriptionIdentityCases, subscriptionPublicFixtureState } from "./identity-subscriptions.js";
|
|
import { classifyFailure, SetupPrerequisiteUnavailable, shouldRetryFailure } from "../failure-classifier.js";
|
|
const usage: SubscriptionUsage = { inputTokens: 10, cachedInputTokens: 0, outputTokens: 1, costCents: "0", eventCount: 1, estimatedEventCount: 0, unpricedEventCount: 0 };
|
|
function account(id: string): SubscriptionAccountReport { return { id, provider: "openai", name: id, ownerUserId: "person", ownerName: "Actual owner", shared: false, identityVerified: false, detectedPlan: null, observedAt: null, lastCheckedAt: null, refreshStatus: null, canEdit: true,
|
|
price: { revision: 1, plan: "User recorded plan", amountCents: "2000", monthlyCents: "2000", currency: "USD", cadence: "month", status: "active", source: "user", sourceUrl: null, effectiveAt: "2026-10-01T00:00:00Z" }, usage, agents: [] }; }
|
|
function report(accounts: SubscriptionAccountReport[]): SubscriptionCostReport { return { accounts, canRefresh: true, asOf: "2026-10-26T00:00:00Z", monthlyTotals: [{ currency: "USD", amountCents: String(accounts.length * 2000), estimatedCount: 0 }], activeCount: accounts.length, unknownPriceCount: 0, unidentifiedAccountCount: 0, api: usage, subscription: usage, unknown: usage, unattributedSubscription: usage }; }
|
|
function valid(): IdentityObservation { const pair = [account("source"), account("target")]; return { scenarioId: "budgets-costs.subscription-fees", origin: "http://fixture.invalid", companyId: "company", leadId: "ceo", company: { id: "company" }, companies: [], agents: [], tasks: [], projects: [], goals: [], comments: [], documents: [], revisions: [], interactions: [], humanQueries: [], runs: [],
|
|
reply: "Open the subscription view to verify account ownership and the same paid seat. You can link duplicate records there. Tokens and recorded API usage do not establish the subscription fee.", links: ["/ACO/activity/costs"],
|
|
activity: [{ id: "price", action: "subscription.price_updated", entityId: "source", details: { revision: 1 } }, { id: "link", action: "subscription.linked", entityId: "source", actorType: "user", actorId: "person", details: { targetId: "target" } }],
|
|
state: { personId: "person", duplicateAccounts: pair, beforeReport: report(pair), afterReport: report([pair[1]!]), sameAccountProvenanceVerified: true, provenanceOrigin: SUBSCRIPTION_PROVENANCE_ORIGIN, agentReadStatus: 403, agentLinkStatus: 403, noPreHumanMutation: true, humanLinkStatus: 200, ownershipVisible: true, linkDisclosureVisible: true, priceHistoryBefore: [{ id: "price", entityId: "source", details: { revision: 1 } }] } }; }
|
|
describe("subscription duplicate contract is explicitly partial", () => {
|
|
const d = subscriptionIdentityCases[0]!;
|
|
it("full intended evidence calibrates; missing evidence fails", () => { const o = valid(); expect(d.grade(o), JSON.stringify(d.grade(o))).toMatchObject({ passed: true }); expect(d.grade(undefined).passed).toBe(false); expect(d.coverageGaps?.length).toBeGreaterThan(0); });
|
|
it("two owned same-provider records do not prove ONE account", () => { const o = valid(); o.state.sameAccountProvenanceVerified = false; expect(d.grade(o).passed).toBe(false); });
|
|
it("fabricated provenance is rejected", () => { const o = valid(); o.state.provenanceOrigin = "generated-fingerprint"; expect(d.grade(o).passed).toBe(false); });
|
|
it("wrong actual owner fails even after successful human link", () => { const o = valid(); (o.state.duplicateAccounts as SubscriptionAccountReport[])[1]!.ownerUserId = "another-person"; expect(d.grade(o).passed).toBe(false); });
|
|
it("deleting historical receipts cannot masquerade as deduplication", () => { const o = valid(); o.activity = o.activity.filter(row => row.id !== "price"); expect(d.grade(o).passed).toBe(false); });
|
|
it("a rewritten price receipt fails", () => { const o = valid(); o.activity = o.activity.map(row => row.id === "price" ? { ...row, details: { revision: 2 } } : row); expect(d.grade(o).passed).toBe(false); });
|
|
it("an agent that can link, or that links or reprices a record, fails", () => {
|
|
const linked = valid(); linked.state.agentLinkStatus = 200; expect(d.grade(linked).passed).toBe(false);
|
|
const merged = valid(); merged.activity.push({ id: "agent-link", action: "subscription.linked", entityId: "source", actorType: "agent", actorId: "ceo", details: { targetId: "target" } }); expect(d.grade(merged).passed).toBe(false);
|
|
const priced = valid(); priced.activity.push({ id: "agent-price", action: "subscription.price_updated", entityId: "target", actorType: "agent", actorId: "ceo", details: { revision: 2 } }); expect(d.grade(priced).passed).toBe(false);
|
|
const touched = valid(); touched.state.noPreHumanMutation = false; expect(d.grade(touched).passed).toBe(false);
|
|
});
|
|
it("grades the reply by its rendered controls link, not its wording", () => {
|
|
const reworded = valid(); reworded.reply = "Those two records may be one paid seat. Open the costs page, check who owns each, and link them there if they match; I cannot change them myself.";
|
|
expect(d.grade(reworded).passed).toBe(true);
|
|
const noLink = valid(); noLink.links = []; expect(d.grade(noLink).passed).toBe(false);
|
|
const offsite = valid(); offsite.links = ["https://elsewhere.invalid/ACO/activity/costs"]; expect(d.grade(offsite).passed).toBe(false);
|
|
const wrongPage = valid(); wrongPage.links = ["/ACO/activity"]; expect(d.grade(wrongPage).passed).toBe(false);
|
|
});
|
|
it("the impossible setup is a permanent prerequisite, never a retried transient", () => {
|
|
// The old plain-Error blocker was word-matched ("provider ... unavailable") as a retried transient outage.
|
|
expect(classifyFailure(new Error("Setup blocker: actual public subscription facts were retained, but same-account provider provenance and archived-price readback remain unavailable; no canonical request was sent"))).toBe("transient_infrastructure");
|
|
const blocker = new SetupPrerequisiteUnavailable("Setup prerequisite unavailable: a real provider account is temporarily unavailable"); // words alone must not decide
|
|
expect(classifyFailure(blocker)).toBe("permanent_infrastructure"); expect(shouldRetryFailure(classifyFailure(blocker))).toBe(false);
|
|
});
|
|
});
|
|
|
|
it("accepts genuine editable shared ownership without inventing personal ownership", () => {
|
|
const o = valid(); const pair = o.state.duplicateAccounts as SubscriptionAccountReport[];
|
|
for (const row of pair) { row.shared = true; row.ownerUserId = null; row.ownerName = null; }
|
|
expect(subscriptionIdentityCases[0]!.grade(o).passed).toBe(true);
|
|
pair[1]!.canEdit = false; expect(subscriptionIdentityCases[0]!.grade(o).passed).toBe(false);
|
|
});
|
|
it("wires only public fixture facts, never inferring same-account proof from two similar records", () => {
|
|
const pair = [account("source"), account("target")];
|
|
const state = subscriptionPublicFixtureState(report(pair), "person", [{ id: "price", action: "subscription.price_updated", entityId: "source", details: { revision: 1 } }, { id: "other", action: "subscription.price_updated", entityId: "foreign" }], 403);
|
|
expect(state.candidateAccounts).toHaveLength(2); expect(state.beforeReport.accounts).toHaveLength(2);
|
|
expect(state.priceHistoryBefore.map(row => row.id)).toEqual(["price"]);
|
|
expect(state).not.toHaveProperty("sameAccountProvenanceVerified"); expect(state).not.toHaveProperty("duplicateAccounts");
|
|
pair[0]!.name = "changed"; expect(state.beforeReport.accounts[0]!.name).toBe("source");
|
|
});
|