mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
172 lines
6.0 KiB
TypeScript
172 lines
6.0 KiB
TypeScript
import { describe, it, expect } from "vitest";
|
|
import type { Page } from "@playwright/test";
|
|
import {
|
|
enterGenericMcpThroughCatalog,
|
|
gradeUnreachableMcp,
|
|
} from "./external-mcp-failure.js";
|
|
import { scopedSessionFailureDiagnostic } from "./external-gateway-client.js";
|
|
import type { ExternalObservation } from "./external-support.js";
|
|
function pageFixture(missing?: string, connectorsInFlyout = false) {
|
|
const actions: string[] = [];
|
|
const page = {
|
|
getByRole: (role: string, options?: { name?: string }) => {
|
|
const name = options?.name ?? "";
|
|
const locator = {
|
|
filter: () => locator,
|
|
first: () => locator,
|
|
isVisible: async () => !(connectorsInFlyout && name === "Connectors" && !actions.includes("More workspace pages")),
|
|
getByRole: (_r: string, o: { name: string }) => ({
|
|
click: async () => {
|
|
if (o.name === missing) throw new Error("Missing real control");
|
|
actions.push(o.name);
|
|
},
|
|
}),
|
|
click: async () => {
|
|
if (name === missing) throw new Error("Missing real control");
|
|
actions.push(name);
|
|
},
|
|
};
|
|
return locator;
|
|
},
|
|
waitForURL: async (accept: (url: URL) => boolean) => {
|
|
expect(accept(new URL("http://fixture.invalid/RUN/apps/byo"))).toBe(true);
|
|
expect(accept(new URL("http://fixture.invalid/RUN/apps"))).toBe(false);
|
|
actions.push("actual /RUN/apps/byo");
|
|
},
|
|
goto: () => {
|
|
throw new Error("Direct navigation rescue is forbidden");
|
|
},
|
|
} as unknown as Page;
|
|
return { page, actions };
|
|
}
|
|
describe("finite held external diagnostics and visible entry correction", () => {
|
|
it("uses visible catalog controls including the expanded custom server option", async () => {
|
|
const { page, actions } = pageFixture();
|
|
await enterGenericMcpThroughCatalog(page, "RUN");
|
|
expect(actions).toEqual([
|
|
"Connectors",
|
|
"Connect",
|
|
"Connect your own MCP server",
|
|
"actual /RUN/apps/byo",
|
|
]);
|
|
});
|
|
it("opens the Coordinator sidebar's More flyout when Connectors is not listed directly", async () => {
|
|
const { page, actions } = pageFixture(undefined, true);
|
|
await enterGenericMcpThroughCatalog(page, "RUN");
|
|
expect(actions).toEqual([
|
|
"More workspace pages",
|
|
"Connectors",
|
|
"Connect",
|
|
"Connect your own MCP server",
|
|
"actual /RUN/apps/byo",
|
|
]);
|
|
});
|
|
it("does not rescue a missing catalog entry with direct navigation", async () => {
|
|
for (const name of [
|
|
"Connectors",
|
|
"Connect",
|
|
"Connect your own MCP server",
|
|
]) {
|
|
const { page, actions } = pageFixture(name);
|
|
await expect(enterGenericMcpThroughCatalog(page, "RUN")).rejects.toThrow(
|
|
"Missing real control",
|
|
);
|
|
expect(actions).not.toContain("actual /RUN/apps/byo");
|
|
}
|
|
});
|
|
const context = {
|
|
companyId: "owned-company",
|
|
agentId: "owned-agent",
|
|
runId: "owned-run",
|
|
issueId: "owned-issue",
|
|
observedRunStatusAtAdmission: "running",
|
|
};
|
|
it("retains actual HTTP500 and admitted run binding without inventing a cause or fixing admission", () => {
|
|
expect(
|
|
scopedSessionFailureDiagnostic(500, context, {
|
|
error: "private diagnostic",
|
|
}),
|
|
).toEqual({
|
|
...context,
|
|
httpStatus: 500,
|
|
reasonCode: "unclassified",
|
|
responseBodyRetained: false,
|
|
});
|
|
});
|
|
it("retains only a recognized policy/context code, never a secret-bearing error or extra response fields", () => {
|
|
for (const reason of [
|
|
"run_company_mismatch",
|
|
"run_agent_mismatch",
|
|
"run_inactive",
|
|
"run_context_mismatch",
|
|
]) {
|
|
const d = scopedSessionFailureDiagnostic(403, context, {
|
|
reasonCode: reason,
|
|
token: "synthetic-credential",
|
|
error: "private body",
|
|
});
|
|
expect(d.reasonCode).toBe(reason);
|
|
expect(JSON.stringify(d)).not.toMatch(
|
|
/synthetic-credential|private body/,
|
|
);
|
|
}
|
|
});
|
|
it("omits unknown, malformed, nested and secret-bearing reason values", () => {
|
|
for (const body of [
|
|
{ reasonCode: "Bearer synthetic-credential" },
|
|
{ reasonCode: "new_upstream_message" },
|
|
{ reasonCode: { token: "synthetic-credential" } },
|
|
{ error: { reasonCode: "run_inactive" } },
|
|
null,
|
|
"private body",
|
|
]) {
|
|
const d = scopedSessionFailureDiagnostic(500, context, body);
|
|
expect(d.reasonCode).toBe("unclassified");
|
|
expect(JSON.stringify(d)).not.toMatch(
|
|
/Bearer|synthetic-credential|private body|new_upstream_message/,
|
|
);
|
|
}
|
|
});
|
|
it("visible-entry correction does not turn the retained model missing-URL question into a completed setup pass", () => {
|
|
const o = {
|
|
scenarioId: "connection-setup.unreachable-mcp",
|
|
companyId: "company",
|
|
origin: "http://fixture.invalid",
|
|
reply: "",
|
|
connections: [],
|
|
phaseReplies: [],
|
|
profiles: [],
|
|
secrets: [],
|
|
endpoints: [],
|
|
interactions: [],
|
|
activity: [],
|
|
links: [],
|
|
runs: [],
|
|
humanQueries: [],
|
|
state: {
|
|
fixtureServerClosed: true,
|
|
serverUrl: "http://127.0.0.1:12345/",
|
|
genericEntryThroughVisibleCatalog: true,
|
|
setupStatus: 502,
|
|
visibleConnectivityError: true,
|
|
checkAgainAvailable: true,
|
|
displayedServerPreserved: true,
|
|
editableUrlPreserved: true,
|
|
abandonedDraft: true,
|
|
noUnexpectedConnection: true,
|
|
},
|
|
} as unknown as ExternalObservation;
|
|
expect(gradeUnreachableMcp(o).passed).toBe(false);
|
|
o.reply =
|
|
"The MCP server is not connected; correct the address or abandon this setup.";
|
|
expect(gradeUnreachableMcp(o).passed).toBe(true);
|
|
o.reply = "That address did not answer. Fix the URL or start the server, then check it again.";
|
|
expect(gradeUnreachableMcp(o).passed).toBe(true);
|
|
o.connections = [{ id: "draft", status: "draft", healthStatus: "healthy" }] as never;
|
|
expect(gradeUnreachableMcp(o).passed).toBe(false);
|
|
o.connections = [];
|
|
o.state.genericEntryThroughVisibleCatalog = false;
|
|
expect(gradeUnreachableMcp(o).passed).toBe(false);
|
|
});
|
|
});
|