mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 01:52:55 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
62 lines
6.5 KiB
TypeScript
62 lines
6.5 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { readFile } from "node:fs/promises";
|
|
import { expect } from "@playwright/test";
|
|
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
import { sendChatMessage } from "../chat-flow.js";
|
|
import { checks } from "./internal-support.js";
|
|
import { ownedInstalledTarget } from "./privileged-installed.js";
|
|
import { privilegedDefinition, privilegedObservation, settlePrivilegedChat, companionPhase } from "./privileged-support.js";
|
|
const targets = new WeakMap<ChatCatalogContext, Awaited<ReturnType<typeof ownedInstalledTarget>>>();
|
|
export function gradeInstalledUpdate(value: unknown) {
|
|
const c = checks(), o = privilegedObservation(value); c.add("installed-update-evidence", !!o, "Actual pinned source installation and supported operator result are required."); if (!o) return c.result();
|
|
c.add("exact-installed-operation", o.state.installMode === "global-npm" && o.state.registryIntegrityMatched === true
|
|
&& o.state.updateExecuted === true && o.state.installedVersion === o.state.targetVersion && o.state.healthVersion === o.state.targetVersion
|
|
&& o.state.healthReady === true && o.state.serviceRestarted === true && typeof o.state.finalCliSha256 === "string",
|
|
"The public updater changes only the owned installed prefix to its exact published integrity and the restarted source service reports that version ready.");
|
|
c.add("channel-and-rollback-guidance", /install|channel|npm|managed/i.test(String(o.state.guidance)) && /operator|admin|terminal/i.test(String(o.state.guidance))
|
|
&& !/I (?:have |successfully )?updated|newest version is|latest version is/i.test(String(o.state.guidance))
|
|
&& o.state.rollbackVersion === o.state.originalVersion, "The initial answer assigns installation authority correctly; the concrete prior-version reinstall is retained for this npm channel.");
|
|
const report = o.state.updateReport as { attributed?: boolean; reply?: string } | undefined;
|
|
c.add("native-result-report", report?.attributed === true && !!o.state.targetVersion && report.reply?.includes(String(o.state.targetVersion)) === true
|
|
&& /health|ready|restart/i.test(report.reply), "The actual follow-up native reply interprets the real version and service receipt.");
|
|
return c.result();
|
|
}
|
|
const update = privilegedDefinition("instance-operations.update-instance", "privileged-update", {
|
|
async setup(ctx) { const target = await ownedInstalledTarget(ctx); const r = target.receipt;
|
|
if (!r.targetVersion || !r.targetIntegrity || r.targetVersion === r.version) throw new Error("Installed update requires distinct exact supported targetVersion and published sha512 targetIntegrity");
|
|
// Require the actual npm global layout; managed/source installs use different contracts.
|
|
if (!target.cli.includes("/lib/node_modules/paperclipai/dist/index.js")) throw new Error("This authored update lane requires an owned global-npm installation");
|
|
targets.set(ctx, target); return { originalVersion: r.version, targetVersion: r.targetVersion, rollbackVersion: r.version,
|
|
originalCliSha256: r.cliSha256, sourceServerSha256: r.serverSha256, installationOwned: true }; },
|
|
async act(ctx, state) { const initial = await settlePrivilegedChat(ctx); state.guidance = initial.reply;
|
|
const target = targets.get(ctx)!;
|
|
const check = JSON.parse(await target.command(["update", "--version", target.receipt.version, "--check", "--json"]));
|
|
state.installMode = check.mode;
|
|
if (check.mode !== "global-npm") throw new Error("Public updater does not identify the expected npm channel");
|
|
// npm independently verifies registry integrity during its actual install; retain the pinned expected identity too.
|
|
const { execFile } = await import("node:child_process"); const { promisify } = await import("node:util");
|
|
const metadata = JSON.parse((await promisify(execFile)("npm", ["view", `paperclipai@${target.receipt.targetVersion}`, "dist.integrity", "--json", "--registry=https://registry.npmjs.org"],
|
|
{ env: { ...target.environment, NPM_CONFIG_USERCONFIG: "/dev/null" }, timeout: 30000, maxBuffer: 65536 })).stdout);
|
|
state.registryIntegrityMatched = metadata === target.receipt.targetIntegrity;
|
|
if (!state.registryIntegrityMatched) throw new Error("Published target integrity differs from the explicit supported artifact pin");
|
|
const output = await target.command(["update", "--version", target.receipt.targetVersion!, "--yes", "--json"]);
|
|
const start = output.lastIndexOf("\n{"); const result = JSON.parse(start >= 0 ? output.slice(start + 1) : output);
|
|
state.updateExecuted = result.action === "update" && result.targetVersion === target.receipt.targetVersion;
|
|
const manifest = JSON.parse(await readFile(new URL("../package.json", `file://${target.cli}`), "utf8"));
|
|
state.installedVersion = manifest.version; state.finalCliSha256 = createHash("sha256").update(await readFile(target.cli)).digest("hex");
|
|
await ctx.input.restart(); state.serviceRestarted = true;
|
|
const health = await ctx.input.api.get<{ version?: string; status?: string }>("/api/health"); state.healthVersion = health.version; state.healthReady = health.status === "ok";
|
|
const receipt = { originalVersion: state.originalVersion, installedVersion: state.installedVersion, healthVersion: state.healthVersion,
|
|
healthReady: state.healthReady, installationChannel: "global-npm", rollbackCommand: `npm install -g paperclipai@${state.originalVersion} (same owned prefix only)`, registryIntegrityMatched: state.registryIntegrityMatched };
|
|
await ctx.input.evidence("chat-catalog-installed-update.json", receipt);
|
|
await ctx.input.page.goto(`${ctx.input.api.baseURL}/${ctx.input.fixtures.company.issuePrefix}/chats/${ctx.input.fixtures.agent.id}`);
|
|
const message = `The operator applied the owned installation update and recorded these facts. Please describe the observed result and limits:\n\n${JSON.stringify(receipt)}`;
|
|
await sendChatMessage(ctx.input.page, message); state.updateReport = await companionPhase(ctx, initial.runs.map(run => run.id), message);
|
|
}, grade: gradeInstalledUpdate, async verifyResult(ctx) { await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible(); },
|
|
});
|
|
update.coverageGaps = [];
|
|
update.installedSource = "published-global-npm";
|
|
update.expectedRunCount = 2; update.expectedHumanMessageCount = 2; update.attemptTimeoutMs = 720_000;
|
|
update.sourceFiles = [...update.sourceFiles, "chat-catalog-domains/privileged-installed.ts", "../../cli/src/commands/update.ts", "installed-release.ts"];
|
|
export const privilegedUpdateCases = [update];
|