Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/task-prerequisite-contracts.test.ts
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

81 lines
8.5 KiB
TypeScript

import { afterEach, beforeEach, expect, it, vi } from "vitest";
import type { Request, Response } from "express";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import { boardMutationGuard } from "../../../server/src/middleware/board-mutation-guard.js";
import { transition } from "../../../server/src/modules/agent-lifecycle/domain/policy.js";
const mock = vi.hoisted(() => ({ gate: vi.fn() }));
vi.mock("./runtime-busy-worker.js", async original => ({ ...await original<object>(), prepareRuntimeWorkerGate: mock.gate }));
import * as inventory from "./internal-task-inventory.js";
import * as delegation from "./runtime-delegation.js";
const companyId = "11111111-1111-4111-8111-111111111111", firstId = "22222222-2222-4222-8222-222222222222", secondId = "33333333-3333-4333-8333-333333333333", leadId = "44444444-4444-4444-8444-444444444444";
const baseURL = "http://127.0.0.1:3218";
function owner(id=firstId) { return { id, companyId, status: "idle", lifecycleState: "ready", lifecycleVersion: 7, lifecycleHolds: [] as string[], lifecycleError: null }; }
function pause(before=owner()) { const requested = { ...before, status: "paused", lifecycleState: transition("ready", "pause"), lifecycleVersion: 8, lifecycleHolds: ["manual"] }; return { before, requested, settled: { ...requested, lifecycleState: transition("pausing", "complete"), lifecycleVersion: 9 } }; }
beforeEach(() => { vi.stubGlobal("fetch", () => { throw new Error("No real network permitted in pure task prerequisites"); }); });
afterEach(() => { vi.useRealTimers(); vi.unstubAllGlobals(); mock.gate.mockReset(); });
it("actual inventory preflight accepts real completed manual pauses instead of impossible ready-plus-paused lifecycle", async () => {
const first=pause(),second=pause(owner(secondId)),rows=[first,second];
const ctx={ input:{fixtures:{company:{id:companyId},agent:{id:leadId}},api:{get:async(path:string)=>path.endsWith(leadId)?owner(leadId):rows.find(x=>path.endsWith(x.before.id))!.settled,getRunObservation:async()=>tasks}}} as unknown as ChatCatalogContext;
const tasks=[{id:"t1",companyId,title:"Launch work 001",description:"d",projectId:"p",status:"blocked",priority:"medium",assigneeAgentId:firstId,goalId:null}];
const state={workerIds:[firstId,secondId],ownerPauseReceipts:rows.map(({before,requested})=>({before,requested})),originalTasks:structuredClone(tasks)};
expect((await inventory.taskInventoryCases[0]!.verifySetup!(ctx,state)).passed).toBe(true);
tasks[0]!.status="todo";
expect((await inventory.taskInventoryCases[0]!.verifySetup!(ctx,state)).checks.find(check=>check.id==="inventory-settled")?.passed).toBe(false);
});
function passesBoardGuard(headers:Record<string,string>) { let passed=false;let status=200;const req={method:"POST",actor:{type:"board",source:"session"},header:(k:string)=>k==="host"?"127.0.0.1:3218":headers[k]??headers[k[0]!.toUpperCase()+k.slice(1)],socket:{},app:{get:()=>undefined}} as unknown as Request;const res={status:(value:number)=>{status=value;return res;},json:()=>{}} as unknown as Response;boardMutationGuard()(req,res,()=>{passed=true;});return{passed,status}; }
it("actual delegation upload uses the owned browser Origin and reaches the unchanged authenticated board guard", async () => {
const stop=new Error("Controlled authenticated upload boundary; no server or file write");mock.gate.mockResolvedValue({worker:{id:firstId},gate:{scriptPath:"/synthetic/gate"}});
const ctx={input:{nonce:"synthetic",fixtures:{company:{id:companyId},agent:{id:leadId}},api:{baseURL,post:async(path:string)=>path.endsWith("/comments")?{id:secondId}:{id:firstId}},page:{url:()=>baseURL+"/RUN/chat",request:{post:async(_url:string,options:{headers?:Record<string,string>})=>{const g=passesBoardGuard(options.headers??{});if(g.passed)throw stop;return{status:()=>g.status,dispose:async()=>{}};}}}},allRuns:async()=>[]} as unknown as ChatCatalogContext;
await expect(delegation.runtimeDelegationCases[0]!.setup(ctx)).rejects.toBe(stop);
});
it("accepts the public pause already-settled response without inventing an operation object", () => {
const r=pause(); expect(inventory.inventoryOwnerPauseRetained({before:r.before,requested:r.settled},r.settled,companyId,firstId)).toBe(true);
});
it.each(["wrong-owner", "foreign-company", "stale-version", "no-hold", "extra-hold", "not-paused", "error", "not-ready-before", "held-before", "invalid-version", "overflow"])("rejects %s inventory pause evidence", kind => {
const r=pause();
if(kind==="wrong-owner")r.settled.id=secondId;
if(kind==="foreign-company")r.settled.companyId=secondId;
if(kind==="stale-version")r.settled.lifecycleVersion--;
if(kind==="no-hold")r.settled.lifecycleHolds=[];
if(kind==="extra-hold")r.settled.lifecycleHolds.push("budget");
if(kind==="not-paused")r.settled.lifecycleState="ready";
if(kind==="error")Object.assign(r.settled,{lifecycleError:"failed"});
if(kind==="not-ready-before")r.before.lifecycleState="paused";
if(kind==="held-before")r.before.lifecycleHolds.push("manual");
if(kind==="invalid-version")r.before.lifecycleVersion=NaN;
if(kind==="overflow")r.before.lifecycleVersion=Number.MAX_SAFE_INTEGER;
expect(inventory.inventoryOwnerPauseRetained(r,r.settled,companyId,firstId)).toBe(false);
});
it("preserves all501 open seeds, both paused owners, genuine unassigned work and assigned in-progress work", () => {
const rows=Array.from({length:501},(_,i)=>inventory.openInventoryTaskSeed(i,firstId,secondId));
expect(rows).toHaveLength(501);expect(rows.filter(r=>r.assigneeAgentId===null)).toHaveLength(167);
expect(rows.some(r=>r.status==="in_progress"&&r.assigneeAgentId===firstId)).toBe(true);
expect(rows.some(r=>r.status==="in_progress"&&r.assigneeAgentId===secondId)).toBe(true);
expect(rows.some(r=>r.status==="blocked")).toBe(true);expect(rows.some(r=>!r.assigneeAgentId&&r.status==="in_progress")).toBe(false);
});
it("requires distinct actual paused owners in the real preflight", async () => {
const r=pause();const ctx={input:{fixtures:{company:{id:companyId},agent:{id:leadId}},api:{get:async(p:string)=>p.endsWith(leadId)?owner(leadId):r.settled,getRunObservation:async()=>[]}}} as unknown as ChatCatalogContext;
const grade=await inventory.taskInventoryCases[0]!.verifySetup!(ctx,{workerIds:[firstId,firstId],ownerPauseReceipts:[r,r]});expect(grade.passed).toBe(false);
});
it.each(["absent", "foreign"])("the actual unchanged board guard rejects %s Origin", kind => {
const headers:Record<string,string>=kind==="absent"?{}:{Origin:"https://foreign.invalid"};expect(passesBoardGuard(headers)).toEqual({passed:false,status:403});
});
it.each(["blank-browser", "foreign-browser", "foreign-upload", "wrong-company", "wrong-task", "query", "credentials"])("rejects %s before human attachment upload", kind => {
let browser=baseURL+"/RUN/chat",target=`/api/companies/${companyId}/issues/${firstId}/attachments`;
if(kind==="blank-browser")browser="about:blank";
if(kind==="foreign-browser")browser="https://foreign.invalid/RUN/chat";
if(kind==="foreign-upload")target="https://foreign.invalid"+target;
if(kind==="wrong-company")target=target.replace(companyId,secondId);
if(kind==="wrong-task")target=target.replace(firstId,secondId);
if(kind==="query")target+="?redirect=1";
if(kind==="credentials")browser="http://user:pass@127.0.0.1:3218/RUN/chat";
expect(()=>delegation.launchBriefBrowserOrigin(baseURL,browser,target,companyId,firstId)).toThrow();
});
it("retains real byte payload, exact owned endpoint and trusted Origin in the caller", async () => {
const stop=new Error("Stop after actual multipart boundary");mock.gate.mockResolvedValue({worker:{id:firstId},gate:{scriptPath:"/synthetic/gate"}});
let calls=0;const post=vi.fn(async(url:string,options:{headers:Record<string,string>;multipart:{issueCommentId:string;file:{name:string;mimeType:string;buffer:Buffer}}})=>{calls++;expect(url).toBe(`/api/companies/${companyId}/issues/${firstId}/attachments`);expect(options.headers).toEqual({Origin:baseURL});expect(options.multipart.issueCommentId).toBe(secondId);expect(options.multipart.file.name).toBe("design-brief.md");expect(options.multipart.file.mimeType).toBe("text/markdown");expect(options.multipart.file.buffer.toString()).toContain("Research without the busywork");throw stop;});
const ctx={input:{nonce:"synthetic",fixtures:{company:{id:companyId},agent:{id:leadId}},api:{baseURL,post:async(p:string)=>p.endsWith("/comments")?{id:secondId}:{id:firstId}},page:{url:()=>baseURL+"/RUN/chat",request:{post}}},allRuns:async()=>[]} as unknown as ChatCatalogContext;
await expect(delegation.runtimeDelegationCases[0]!.setup(ctx)).rejects.toBe(stop);expect(calls).toBe(1);
});