mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog checks that a chat agent does the work a person asks for > - In the latest full-cohort run on master, 16 cases failed although the saved work was correct > - Their graders matched words, one literal value, or one API route in free-form model output > - A grader that fails correct work hides real failures in the noise, and it can also pass wrong work that uses the expected word > - This pull request grades those cases by saved state, audit records, setup literals and rendered page structure, and keeps only narrow checks for false claims > - The benefit is that these cases fail only when the product or agent does the wrong thing ## Linked Issues or Issue Description Refs #18971 (related: company-secret question handoffs) **What happened** The full-cohort chat-eval run on master failed 16 cases where the retained evidence shows correct behaviour. Examples: `companies.rename-company` accepted only `company.updated`, but the supported branding route logs `company.branding_updated`. `recovery.post-stop-message` required a signup button label that matched a regex, and rejected a labelled email form with the button "Start writing free". `secrets.secret-proposal` flagged "the value is delivered when Morgan next runs" as a false delivery claim. Other graders required words such as "uncommitted", "telemetry", "administrator" or the app name that the question did not ask for. **Expected behavior** A grader accepts every supported route and wording that achieves the outcome. It fails when saved state is wrong, when a requested fact is wrong, or when the reply makes a claim that saved state contradicts. **Steps to reproduce** 1. Load the retained observation for one of the cases below from the full-cohort evidence. 2. Grade it with the old oracle. The keyword or route check fails although the saved state is correct. 3. Grade it with the new oracle. It passes. A broken variant (wrong saved state or a contradicting claim) still fails. ## What Changed - `activity.audit-change`: attribution is graded by structure. The reply names the recorded editor, names no other company agent (new setup literal `otherActors`), and states only real transitions. Sentence grammar is no longer matched. - `app-permissions.ask-before-action`: each continuation reply must exist and must link no app record other than the reviewed connection. It no longer must name the app. - `budgets-costs.spend-breakdown`: the reply must point at the record with unresolved usage (task link, identifier or run id) and must not give it a zero or free amount. The words "subscription" and "incomplete" are no longer required. - `cases.browse-cases`: an excluded case may be mentioned in prose. It must not be linked or listed as a result row. - `cases.handoff-same-task`: handoff instructions carry findings forward when they point at the saved `findings` document, reproduce it, or restate it. The premise (a primary Morgan run performs the handoff) is unchanged. - `cases.update-case-facts`: a saved browser value may add clauses that only rule out other platforms ("desktop Safari unaffected"). - `companies.agent-observability`: the reply no longer must name Telemetry and the run log. It must not claim that they are exported. - `companies.rename-company`: both supported name routes (`company.updated`, `company.branding_updated`) are accepted. - `connection-setup.choose-provider`: the answer must exist and must not claim a connected account. The unavailable method no longer must be named; the setup path stays graded by clicking the card or link. - `goals.align-existing-work`: a question card from the answering run counts as the phase-two response. Refusal words and "unchanged" words are no longer required; false change claims still fail. - `recovery.post-stop-message` (and `steering.interrupt-with-direction`, which shares the helper): a visible form with a labelled field and a submit control is a signup action, whatever the button says. - `routines.pause-owned`: the "paused"/"running" words are no longer required. The saved checkpoint proves both; a cancellation claim still fails. - `secrets.secret-proposal` (and `secrets.configure-company-secret`, which shares the helper): a sentence that says when delivery will happen is not a delivery claim. - `skills.explicit-restriction`: the reply no longer must say "administrator". Success claims and invented tool limitations still fail. - `workspaces.inspect-git`: the reply no longer must say "uncommitted". Unchanged HEAD and status prove it; commit or push claims still fail. - Every affected `oracleVersion` is bumped in code and in `chat-catalog-descriptors.json`. No scenario YAML changed. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. - `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact canary with Chromium: pass, including new form cases. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass. `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged pins. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Offline calibration on retained evidence, for each case: the retained correct evidence passes, earlier passing evidence passes where it exists, and a broken variant fails. `spend-breakdown` retained evidence still fails, because its reply shows unresolved usage as `$0.00`. `handoff-same-task` retained evidence still fails on the kept primary-run premise. - Live paid runs: see the comment below. ## Risks - Low risk. The change is limited to eval graders and their unit tests. Product code does not change. - Each grader is more accepting of wording. Saved-state checks and the narrow false-claim checks remain, so a wrong outcome still fails. - `secrets.configure-company-secret` still fails its retained evidence. Its agent asked a two-question card, and the act supports only one-question cards. This pull request changes it only by the shared delivery-claim helper. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
28 lines
3.2 KiB
TypeScript
28 lines
3.2 KiB
TypeScript
import { it, expect } from "vitest";
|
|
import { gradeInspectGit } from "./runtime-git.js";
|
|
function valid() {
|
|
const task = { id: "website", title: "Website task", status: "done", assigneeAgentId: "morgan" };
|
|
const workspaceBefore = { workspace: { id: "actual", cwd: "/isolated/actual" } };
|
|
const diff = "diff --git a/header.html b/header.html\n-Launch later\n+Launch sooner\ndiff --git a/existing-notes.txt b/existing-notes.txt\n+Existing unrelated human amendment";
|
|
return { scenarioId: "workspaces.inspect-git", origin: "http://fixture.invalid", companyId: "company", leadId: "ceo",
|
|
state: { task, workspaceBefore, workspaceAfter: structuredClone(workspaceBefore), nativeInspectionCwd: "/isolated/actual", noRemote: true,
|
|
diffBefore: diff, diffAfter: diff, statusBefore: " M header.html\n M existing-notes.txt", statusAfter: " M header.html\n M existing-notes.txt",
|
|
headBefore: "baseline", headAfter: "baseline", headerBefore: "Launch sooner", headerAfter: "Launch sooner", notesBefore: "Amended notes", notesAfter: "Amended notes" },
|
|
agents: [], tasks: [task], links: [], activity: [], humanQueries: [],
|
|
reply: "The uncommitted change in header.html replaces Launch later with Launch sooner. existing-notes.txt also has the pre-existing human amendment. These remain working tree changes.",
|
|
runs: [{ id: "setup", agentId: "morgan", status: "succeeded" }, { id: "query", agentId: "ceo", status: "succeeded" }] };
|
|
}
|
|
it("accepts an actual uncommitted diff summary with unchanged prior work", () => expect(gradeInspectGit(valid()).passed).toBe(true));
|
|
it("accepts natural substantive working-tree reporting", () => { const o = valid(); o.reply = "header.html has an uncommitted replacement: Launch later becomes Launch sooner. The working tree also retains the human edit in existing-notes.txt."; expect(gradeInspectGit(o).passed).toBe(true); });
|
|
it("rejects absent repository evidence", () => expect(gradeInspectGit(undefined).passed).toBe(false));
|
|
it("rejects inspecting another checkout", () => { const o = valid(); o.state.nativeInspectionCwd = "/other/website"; expect(gradeInspectGit(o).passed).toBe(false); });
|
|
it("rejects dropping the unrelated existing change", () => { const o = valid(); o.state.notesAfter = ""; expect(gradeInspectGit(o).passed).toBe(false); });
|
|
it("rejects committing while claiming to inspect", () => { const o = valid(); o.state.headAfter = "new-commit"; expect(gradeInspectGit(o).passed).toBe(false); });
|
|
it("rejects invented remote publication", () => { const o = valid(); o.reply += " I pushed the changes."; expect(gradeInspectGit(o).passed).toBe(false); });
|
|
it("rejects a plausible summary with no real observed diff", () => { const o = valid(); o.state.diffBefore = ""; o.state.diffAfter = ""; expect(gradeInspectGit(o).passed).toBe(false); });
|
|
it("does not require the word uncommitted when HEAD and status prove it, but still rejects a commit claim", () => {
|
|
const o = valid(); o.reply = "Website file changes vs. the baseline:\n- `header.html`: heading changed from `<h1>Launch later</h1>` to `<h1>Launch sooner</h1>`.\n\nAlso modified, but not a website file: `existing-notes.txt`.";
|
|
expect(gradeInspectGit(o).passed).toBe(true);
|
|
o.reply += "\nI've committed these changes."; expect(gradeInspectGit(o).passed).toBe(false);
|
|
});
|