Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/runtime-git.test.ts
T
DottaandClaude Opus 5.5 5f34f777ec fix(evals): grade 16 chat cases by saved state instead of reply keywords (#20600)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog checks that a chat agent does the work
a person asks for
> - In the latest full-cohort run on master, 16 cases failed although
the saved work was correct
> - Their graders matched words, one literal value, or one API route in
free-form model output
> - A grader that fails correct work hides real failures in the noise,
and it can also pass wrong work that uses the expected word
> - This pull request grades those cases by saved state, audit records,
setup literals and rendered page structure, and keeps only narrow checks
for false claims
> - The benefit is that these cases fail only when the product or agent
does the wrong thing

## Linked Issues or Issue Description

Refs #18971 (related: company-secret question handoffs)

**What happened**

The full-cohort chat-eval run on master failed 16 cases where the
retained evidence shows correct behaviour. Examples:
`companies.rename-company` accepted only `company.updated`, but the
supported branding route logs `company.branding_updated`.
`recovery.post-stop-message` required a signup button label that matched
a regex, and rejected a labelled email form with the button "Start
writing free". `secrets.secret-proposal` flagged "the value is delivered
when Morgan next runs" as a false delivery claim. Other graders required
words such as "uncommitted", "telemetry", "administrator" or the app
name that the question did not ask for.

**Expected behavior**

A grader accepts every supported route and wording that achieves the
outcome. It fails when saved state is wrong, when a requested fact is
wrong, or when the reply makes a claim that saved state contradicts.

**Steps to reproduce**

1. Load the retained observation for one of the cases below from the
full-cohort evidence.
2. Grade it with the old oracle. The keyword or route check fails
although the saved state is correct.
3. Grade it with the new oracle. It passes. A broken variant (wrong
saved state or a contradicting claim) still fails.

## What Changed

- `activity.audit-change`: attribution is graded by structure. The reply
names the recorded editor, names no other company agent (new setup
literal `otherActors`), and states only real transitions. Sentence
grammar is no longer matched.
- `app-permissions.ask-before-action`: each continuation reply must
exist and must link no app record other than the reviewed connection. It
no longer must name the app.
- `budgets-costs.spend-breakdown`: the reply must point at the record
with unresolved usage (task link, identifier or run id) and must not
give it a zero or free amount. The words "subscription" and "incomplete"
are no longer required.
- `cases.browse-cases`: an excluded case may be mentioned in prose. It
must not be linked or listed as a result row.
- `cases.handoff-same-task`: handoff instructions carry findings forward
when they point at the saved `findings` document, reproduce it, or
restate it. The premise (a primary Morgan run performs the handoff) is
unchanged.
- `cases.update-case-facts`: a saved browser value may add clauses that
only rule out other platforms ("desktop Safari unaffected").
- `companies.agent-observability`: the reply no longer must name
Telemetry and the run log. It must not claim that they are exported.
- `companies.rename-company`: both supported name routes
(`company.updated`, `company.branding_updated`) are accepted.
- `connection-setup.choose-provider`: the answer must exist and must not
claim a connected account. The unavailable method no longer must be
named; the setup path stays graded by clicking the card or link.
- `goals.align-existing-work`: a question card from the answering run
counts as the phase-two response. Refusal words and "unchanged" words
are no longer required; false change claims still fail.
- `recovery.post-stop-message` (and `steering.interrupt-with-direction`,
which shares the helper): a visible form with a labelled field and a
submit control is a signup action, whatever the button says.
- `routines.pause-owned`: the "paused"/"running" words are no longer
required. The saved checkpoint proves both; a cancellation claim still
fails.
- `secrets.secret-proposal` (and `secrets.configure-company-secret`,
which shares the helper): a sentence that says when delivery will happen
is not a delivery claim.
- `skills.explicit-restriction`: the reply no longer must say
"administrator". Success claims and invented tool limitations still
fail.
- `workspaces.inspect-git`: the reply no longer must say "uncommitted".
Unchanged HEAD and status prove it; commit or push claims still fail.
- Every affected `oracleVersion` is bumped in code and in
`chat-catalog-descriptors.json`. No scenario YAML changed.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000 tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass.
- `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact
canary with Chromium: pass, including new form cases.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged pins.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- Offline calibration on retained evidence, for each case: the retained
correct evidence passes, earlier passing evidence passes where it
exists, and a broken variant fails. `spend-breakdown` retained evidence
still fails, because its reply shows unresolved usage as `$0.00`.
`handoff-same-task` retained evidence still fails on the kept
primary-run premise.
- Live paid runs: see the comment below.

## Risks

- Low risk. The change is limited to eval graders and their unit tests.
Product code does not change.
- Each grader is more accepting of wording. Saved-state checks and the
narrow false-claim checks remain, so a wrong outcome still fails.
- `secrets.configure-company-secret` still fails its retained evidence.
Its agent asked a two-question card, and the act supports only
one-question cards. This pull request changes it only by the shared
delivery-claim helper.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:37:16 -06:00

28 lines
3.2 KiB
TypeScript

import { it, expect } from "vitest";
import { gradeInspectGit } from "./runtime-git.js";
function valid() {
const task = { id: "website", title: "Website task", status: "done", assigneeAgentId: "morgan" };
const workspaceBefore = { workspace: { id: "actual", cwd: "/isolated/actual" } };
const diff = "diff --git a/header.html b/header.html\n-Launch later\n+Launch sooner\ndiff --git a/existing-notes.txt b/existing-notes.txt\n+Existing unrelated human amendment";
return { scenarioId: "workspaces.inspect-git", origin: "http://fixture.invalid", companyId: "company", leadId: "ceo",
state: { task, workspaceBefore, workspaceAfter: structuredClone(workspaceBefore), nativeInspectionCwd: "/isolated/actual", noRemote: true,
diffBefore: diff, diffAfter: diff, statusBefore: " M header.html\n M existing-notes.txt", statusAfter: " M header.html\n M existing-notes.txt",
headBefore: "baseline", headAfter: "baseline", headerBefore: "Launch sooner", headerAfter: "Launch sooner", notesBefore: "Amended notes", notesAfter: "Amended notes" },
agents: [], tasks: [task], links: [], activity: [], humanQueries: [],
reply: "The uncommitted change in header.html replaces Launch later with Launch sooner. existing-notes.txt also has the pre-existing human amendment. These remain working tree changes.",
runs: [{ id: "setup", agentId: "morgan", status: "succeeded" }, { id: "query", agentId: "ceo", status: "succeeded" }] };
}
it("accepts an actual uncommitted diff summary with unchanged prior work", () => expect(gradeInspectGit(valid()).passed).toBe(true));
it("accepts natural substantive working-tree reporting", () => { const o = valid(); o.reply = "header.html has an uncommitted replacement: Launch later becomes Launch sooner. The working tree also retains the human edit in existing-notes.txt."; expect(gradeInspectGit(o).passed).toBe(true); });
it("rejects absent repository evidence", () => expect(gradeInspectGit(undefined).passed).toBe(false));
it("rejects inspecting another checkout", () => { const o = valid(); o.state.nativeInspectionCwd = "/other/website"; expect(gradeInspectGit(o).passed).toBe(false); });
it("rejects dropping the unrelated existing change", () => { const o = valid(); o.state.notesAfter = ""; expect(gradeInspectGit(o).passed).toBe(false); });
it("rejects committing while claiming to inspect", () => { const o = valid(); o.state.headAfter = "new-commit"; expect(gradeInspectGit(o).passed).toBe(false); });
it("rejects invented remote publication", () => { const o = valid(); o.reply += " I pushed the changes."; expect(gradeInspectGit(o).passed).toBe(false); });
it("rejects a plausible summary with no real observed diff", () => { const o = valid(); o.state.diffBefore = ""; o.state.diffAfter = ""; expect(gradeInspectGit(o).passed).toBe(false); });
it("does not require the word uncommitted when HEAD and status prove it, but still rejects a commit claim", () => {
const o = valid(); o.reply = "Website file changes vs. the baseline:\n- `header.html`: heading changed from `<h1>Launch later</h1>` to `<h1>Launch sooner</h1>`.\n\nAlso modified, but not a website file: `existing-notes.txt`.";
expect(gradeInspectGit(o).passed).toBe(true);
o.reply += "\nI've committed these changes."; expect(gradeInspectGit(o).passed).toBe(false);
});