mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 08:53:19 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
81 lines
8.5 KiB
TypeScript
81 lines
8.5 KiB
TypeScript
import { afterEach, beforeEach, expect, it, vi } from "vitest";
|
|
import type { Request, Response } from "express";
|
|
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
import { boardMutationGuard } from "../../../server/src/middleware/board-mutation-guard.js";
|
|
import { transition } from "../../../server/src/modules/agent-lifecycle/domain/policy.js";
|
|
const mock = vi.hoisted(() => ({ gate: vi.fn() }));
|
|
vi.mock("./runtime-busy-worker.js", async original => ({ ...await original<object>(), prepareRuntimeWorkerGate: mock.gate }));
|
|
import * as inventory from "./internal-task-inventory.js";
|
|
import * as delegation from "./runtime-delegation.js";
|
|
const companyId = "11111111-1111-4111-8111-111111111111", firstId = "22222222-2222-4222-8222-222222222222", secondId = "33333333-3333-4333-8333-333333333333", leadId = "44444444-4444-4444-8444-444444444444";
|
|
const baseURL = "http://127.0.0.1:3218";
|
|
function owner(id=firstId) { return { id, companyId, status: "idle", lifecycleState: "ready", lifecycleVersion: 7, lifecycleHolds: [] as string[], lifecycleError: null }; }
|
|
function pause(before=owner()) { const requested = { ...before, status: "paused", lifecycleState: transition("ready", "pause"), lifecycleVersion: 8, lifecycleHolds: ["manual"] }; return { before, requested, settled: { ...requested, lifecycleState: transition("pausing", "complete"), lifecycleVersion: 9 } }; }
|
|
beforeEach(() => { vi.stubGlobal("fetch", () => { throw new Error("No real network permitted in pure task prerequisites"); }); });
|
|
afterEach(() => { vi.useRealTimers(); vi.unstubAllGlobals(); mock.gate.mockReset(); });
|
|
it("actual inventory preflight accepts real completed manual pauses instead of impossible ready-plus-paused lifecycle", async () => {
|
|
const first=pause(),second=pause(owner(secondId)),rows=[first,second];
|
|
const ctx={ input:{fixtures:{company:{id:companyId},agent:{id:leadId}},api:{get:async(path:string)=>path.endsWith(leadId)?owner(leadId):rows.find(x=>path.endsWith(x.before.id))!.settled,getRunObservation:async()=>tasks}}} as unknown as ChatCatalogContext;
|
|
const tasks=[{id:"t1",companyId,title:"Launch work 001",description:"d",projectId:"p",status:"blocked",priority:"medium",assigneeAgentId:firstId,goalId:null}];
|
|
const state={workerIds:[firstId,secondId],ownerPauseReceipts:rows.map(({before,requested})=>({before,requested})),originalTasks:structuredClone(tasks)};
|
|
expect((await inventory.taskInventoryCases[0]!.verifySetup!(ctx,state)).passed).toBe(true);
|
|
tasks[0]!.status="todo";
|
|
expect((await inventory.taskInventoryCases[0]!.verifySetup!(ctx,state)).checks.find(check=>check.id==="inventory-settled")?.passed).toBe(false);
|
|
});
|
|
function passesBoardGuard(headers:Record<string,string>) { let passed=false;let status=200;const req={method:"POST",actor:{type:"board",source:"session"},header:(k:string)=>k==="host"?"127.0.0.1:3218":headers[k]??headers[k[0]!.toUpperCase()+k.slice(1)],socket:{},app:{get:()=>undefined}} as unknown as Request;const res={status:(value:number)=>{status=value;return res;},json:()=>{}} as unknown as Response;boardMutationGuard()(req,res,()=>{passed=true;});return{passed,status}; }
|
|
it("actual delegation upload uses the owned browser Origin and reaches the unchanged authenticated board guard", async () => {
|
|
const stop=new Error("Controlled authenticated upload boundary; no server or file write");mock.gate.mockResolvedValue({worker:{id:firstId},gate:{scriptPath:"/synthetic/gate"}});
|
|
const ctx={input:{nonce:"synthetic",fixtures:{company:{id:companyId},agent:{id:leadId}},api:{baseURL,post:async(path:string)=>path.endsWith("/comments")?{id:secondId}:{id:firstId}},page:{url:()=>baseURL+"/RUN/chat",request:{post:async(_url:string,options:{headers?:Record<string,string>})=>{const g=passesBoardGuard(options.headers??{});if(g.passed)throw stop;return{status:()=>g.status,dispose:async()=>{}};}}}},allRuns:async()=>[]} as unknown as ChatCatalogContext;
|
|
await expect(delegation.runtimeDelegationCases[0]!.setup(ctx)).rejects.toBe(stop);
|
|
});
|
|
|
|
it("accepts the public pause already-settled response without inventing an operation object", () => {
|
|
const r=pause(); expect(inventory.inventoryOwnerPauseRetained({before:r.before,requested:r.settled},r.settled,companyId,firstId)).toBe(true);
|
|
});
|
|
it.each(["wrong-owner", "foreign-company", "stale-version", "no-hold", "extra-hold", "not-paused", "error", "not-ready-before", "held-before", "invalid-version", "overflow"])("rejects %s inventory pause evidence", kind => {
|
|
const r=pause();
|
|
if(kind==="wrong-owner")r.settled.id=secondId;
|
|
if(kind==="foreign-company")r.settled.companyId=secondId;
|
|
if(kind==="stale-version")r.settled.lifecycleVersion--;
|
|
if(kind==="no-hold")r.settled.lifecycleHolds=[];
|
|
if(kind==="extra-hold")r.settled.lifecycleHolds.push("budget");
|
|
if(kind==="not-paused")r.settled.lifecycleState="ready";
|
|
if(kind==="error")Object.assign(r.settled,{lifecycleError:"failed"});
|
|
if(kind==="not-ready-before")r.before.lifecycleState="paused";
|
|
if(kind==="held-before")r.before.lifecycleHolds.push("manual");
|
|
if(kind==="invalid-version")r.before.lifecycleVersion=NaN;
|
|
if(kind==="overflow")r.before.lifecycleVersion=Number.MAX_SAFE_INTEGER;
|
|
expect(inventory.inventoryOwnerPauseRetained(r,r.settled,companyId,firstId)).toBe(false);
|
|
});
|
|
it("preserves all501 open seeds, both paused owners, genuine unassigned work and assigned in-progress work", () => {
|
|
const rows=Array.from({length:501},(_,i)=>inventory.openInventoryTaskSeed(i,firstId,secondId));
|
|
expect(rows).toHaveLength(501);expect(rows.filter(r=>r.assigneeAgentId===null)).toHaveLength(167);
|
|
expect(rows.some(r=>r.status==="in_progress"&&r.assigneeAgentId===firstId)).toBe(true);
|
|
expect(rows.some(r=>r.status==="in_progress"&&r.assigneeAgentId===secondId)).toBe(true);
|
|
expect(rows.some(r=>r.status==="blocked")).toBe(true);expect(rows.some(r=>!r.assigneeAgentId&&r.status==="in_progress")).toBe(false);
|
|
});
|
|
it("requires distinct actual paused owners in the real preflight", async () => {
|
|
const r=pause();const ctx={input:{fixtures:{company:{id:companyId},agent:{id:leadId}},api:{get:async(p:string)=>p.endsWith(leadId)?owner(leadId):r.settled,getRunObservation:async()=>[]}}} as unknown as ChatCatalogContext;
|
|
const grade=await inventory.taskInventoryCases[0]!.verifySetup!(ctx,{workerIds:[firstId,firstId],ownerPauseReceipts:[r,r]});expect(grade.passed).toBe(false);
|
|
});
|
|
it.each(["absent", "foreign"])("the actual unchanged board guard rejects %s Origin", kind => {
|
|
const headers:Record<string,string>=kind==="absent"?{}:{Origin:"https://foreign.invalid"};expect(passesBoardGuard(headers)).toEqual({passed:false,status:403});
|
|
});
|
|
it.each(["blank-browser", "foreign-browser", "foreign-upload", "wrong-company", "wrong-task", "query", "credentials"])("rejects %s before human attachment upload", kind => {
|
|
let browser=baseURL+"/RUN/chat",target=`/api/companies/${companyId}/issues/${firstId}/attachments`;
|
|
if(kind==="blank-browser")browser="about:blank";
|
|
if(kind==="foreign-browser")browser="https://foreign.invalid/RUN/chat";
|
|
if(kind==="foreign-upload")target="https://foreign.invalid"+target;
|
|
if(kind==="wrong-company")target=target.replace(companyId,secondId);
|
|
if(kind==="wrong-task")target=target.replace(firstId,secondId);
|
|
if(kind==="query")target+="?redirect=1";
|
|
if(kind==="credentials")browser="http://user:pass@127.0.0.1:3218/RUN/chat";
|
|
expect(()=>delegation.launchBriefBrowserOrigin(baseURL,browser,target,companyId,firstId)).toThrow();
|
|
});
|
|
it("retains real byte payload, exact owned endpoint and trusted Origin in the caller", async () => {
|
|
const stop=new Error("Stop after actual multipart boundary");mock.gate.mockResolvedValue({worker:{id:firstId},gate:{scriptPath:"/synthetic/gate"}});
|
|
let calls=0;const post=vi.fn(async(url:string,options:{headers:Record<string,string>;multipart:{issueCommentId:string;file:{name:string;mimeType:string;buffer:Buffer}}})=>{calls++;expect(url).toBe(`/api/companies/${companyId}/issues/${firstId}/attachments`);expect(options.headers).toEqual({Origin:baseURL});expect(options.multipart.issueCommentId).toBe(secondId);expect(options.multipart.file.name).toBe("design-brief.md");expect(options.multipart.file.mimeType).toBe("text/markdown");expect(options.multipart.file.buffer.toString()).toContain("Research without the busywork");throw stop;});
|
|
const ctx={input:{nonce:"synthetic",fixtures:{company:{id:companyId},agent:{id:leadId}},api:{baseURL,post:async(p:string)=>p.endsWith("/comments")?{id:secondId}:{id:firstId}},page:{url:()=>baseURL+"/RUN/chat",request:{post}}},allRuns:async()=>[]} as unknown as ChatCatalogContext;
|
|
await expect(delegation.runtimeDelegationCases[0]!.setup(ctx)).rejects.toBe(stop);expect(calls).toBe(1);
|
|
});
|