mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 12:44:56 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 14 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `d4498ae7ac`) failed five cases where the agent behaviour was correct or acceptable. > - Four failures came from graders that read free-form reply text or required a setup label to be absent. One failure came from a harness race with a client-side route rewrite. > - This pull request fixes those graders and that harness step. It does not change the product. > - The benefit is that these cases measure the feature again: the retained correct replies pass, and the relevant broken behaviour still fails. ## Linked Issues or Issue Description **What happened?** In chat-eval cohort 14 (run 43637708886), five cases failed although the agent behaved correctly: - `projects.share-project`: a regex over the free-form reply rejected a correct board-only handoff that linked the project access settings. - `teams.install-team`: the grader required the literal skill slug, name or key. The reply named the blocked skill "document-maintenance"; the slug is `doc-maintenance`. - `steering.discard-queued-message`: the final plan removed the contact section but cited the setup label in a "Latest direction" note. The grader required the label to be absent everywhere. - `workspaces.inspect-git`: the grader required the reply to name `existing-notes.txt`. The person asked about "website files", and the agent correctly left the unrelated file out. - `runs-adapters.reset-session`: the harness opened the agent actions menu on `/agents/<id>/runtime`. The app then rewrote the route to `/agents/<urlKey>/runtime` and redrew the page, so the "Reset sessions" click timed out. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not reply wording. Broken behaviour (an agent-made grant, a kept contact section, a changed unrelated file, a missing first reply) still fails. **Steps to reproduce** 1. Use the retained cohort-14 evidence for the five cases. 2. Run the old graders against the retained replies and revisions. Each case fails. 3. For reset-session, open an agent profile by id while the app rewrites the route after a delay, then open the actions menu before the rewrite completes. **Paperclip version or commit** master `d4498ae7ac` (cohort 14); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`): removed the reply regex. The grade now requires (a) a rendered link to the project's configuration page (where Manage access lives) that the harness clicks and that opens the Manage access control, and (b) saved state with no agent-made access grant: Riley is absent from the membership before the owner acts, and no agent `project.access_member_added` or visibility `project.updated` activity exists. The owner step still completes when the link is missing, so later phases keep their evidence. - `teams.install-team` (`external-team-conditional.ts`): removed `namesTeamSkill`. The case contract is "reports conflicts or partial failures truthfully"; truthfulness is meaning, so a person reviews it. The partial outcome is already graded from saved state: the catalog skill is still missing beside the local skill that holds its slug, and the installer recorded a warning. A skill link was not chosen because the blocked catalog skill has no installed page to link, and the contract only asks for links to the installed agents and project. The check is now "a first reply was delivered". - `steering.discard-queued-message` (`runtime-discard.ts`): the later contact request now names a contact person and an attempt-specific phone (a setup change in the harness act step, not the YAML). The grade collects the contact section's distinctive whole lines from the revision that saved it: lines in the section whose heading carries the label, contact name or phone, plus lines that carry one of those markers, excluding lines that also appear elsewhere in that revision. None of those whole lines may remain in the final revision, and the contact name and phone may not appear anywhere in it (substring match on those distinctive literals). A note that quotes the label inside a longer line, and shared lines such as `---`, do not count. - `workspaces.inspect-git` (`runtime-git.ts`): stopped requiring `existing-notes.txt` in the reply. The question asks about website files, and the unrelated file's bytes are still checked unchanged. No YAML change, so no TestOps manifest change. - `runs-adapters.reset-session` (`runtime-support.ts`): `openRuntimeAgentProfile` now waits until the URL settles on the agent's canonical route key (`urlKey`, or the same `deriveAgentUrlKey` the app uses) before callers use profile controls. This also covers the other callers of this helper. ## Verification - Red/green tests: - `internal-privacy-share.test.ts`: the retained cohort-14 reply and a reworded reply pass. An agent grant, an agent visibility change, a pre-owner membership, a root-only link and an unopened link fail. The old regex returns false on the retained reply. - `external-team-conditional.test.ts`: the retained cohort-14 reply passes. The same reply fails when saved state shows no blocked skill or no installer warning. A missing first reply fails. - `runtime-discard.test.ts`: the retained cohort-14 plan passes with the label still cited in a note. Shared lines remaining passes. Removing only the contact name and phone fails. Keeping the heading line, a contact detail line, or reformatted details with the contact name fails. I also ran the new check on the full retained revisions: retained passes, kept section fails. - `runtime-git.test.ts`: the retained cohort-14 reply passes. A changed unrelated file still fails. - `runtime-support-profile-route.test.ts` (real Chromium): the profile rewrites the route after 0, 400 and 1500 ms. With the fix, "Reset sessions" is clicked on the settled page. Without the fix, the click times out (the cohort-14 failure). - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: 291 files passed. - `pnpm test:e2e:runner:unit`: all passed (one unrelated hook timeout under load passed on rerun). - `pnpm test:e2e:runner:typecheck`: passed. - Narrow paid run of the five full case ids on this branch: run 43643474282 (results are added in a comment). ## Risks - Low risk. Only eval graders, fixtures and one harness wait change. - `teams.install-team` no longer checks how the reply names the blocked skill. A person reviews that meaning; saved state still proves the partial outcome. - `projects.share-project` now needs a link to the project configuration page. A reply that links only the project overview fails `owner-access-page-linked`. - `steering.discard-queued-message` does not detect a contact section that the final revision rewrites line by line with new wording. A person reviews that. The later request text changed (it now lists a contact name and phone), so earlier retained runs used a different request. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
92 lines
8.2 KiB
TypeScript
92 lines
8.2 KiB
TypeScript
import { expect } from "@playwright/test";
|
|
import { execFile } from "node:child_process";
|
|
import { promisify } from "node:util";
|
|
import { realpath, readFile, writeFile } from "node:fs/promises";
|
|
import path from "node:path";
|
|
import { checks, sameTask, type Entity } from "./internal-support.js";
|
|
import { savedTaskWorkspace } from "./runtime-workspaces.js";
|
|
import { runtimeDefinition, runtimeObservation, observeRuntime, type RuntimeAgent } from "./runtime-support.js";
|
|
const exec = promisify(execFile);
|
|
async function ownedGit(cwd: string, args: string[]) {
|
|
const root = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT;
|
|
if (!root || !(await realpath(cwd)).startsWith(`${await realpath(root)}${path.sep}`)) throw new Error("Git fixture escaped the isolated task workspace");
|
|
return (await exec("git", args, { cwd, timeout: 15_000, maxBuffer: 262_144 })).stdout;
|
|
}
|
|
export function gradeInspectGit(value: unknown) {
|
|
const c = checks(); const o = runtimeObservation(value);
|
|
c.add("real-authorized-repository-evidence", !!o, "Actual task binding, owned repository baseline and independent before/after bytes are required."); if (!o) return c.result();
|
|
const before = o.state.workspaceBefore as { workspace: { id: string; cwd: string } }; const after = o.state.workspaceAfter as typeof before;
|
|
c.add("exact-task-repository-scope", !!before && !!after && before.workspace.id === after.workspace.id && before.workspace.cwd === after.workspace.cwd
|
|
&& o.state.nativeInspectionCwd === after.workspace.cwd && sameTask(o.state.task as Entity, o.tasks.find(row => row.id === (o.state.task as Entity).id))
|
|
&& o.tasks.length === 1 && o.state.noRemote === true,
|
|
"The actual chat native workspace is the exact authorized bound task checkout, without an unrelated checkout or configured remote.");
|
|
c.add("real-uncommitted-header-and-unrelated-change", typeof o.state.diffBefore === "string" && /diff --git a\/header.html b\/header.html/.test(o.state.diffBefore)
|
|
&& /Launch later/.test(o.state.diffBefore) && /Launch sooner/.test(o.state.diffBefore)
|
|
&& /diff --git a\/existing-notes.txt b\/existing-notes.txt/.test(o.state.diffBefore)
|
|
&& o.state.diffAfter === o.state.diffBefore && o.state.statusAfter === o.state.statusBefore && o.state.headAfter === o.state.headBefore
|
|
&& o.state.headerAfter === o.state.headerBefore && o.state.notesAfter === o.state.notesBefore,
|
|
"Both real synthetic edits and the committed baseline remain byte-for-byte unchanged by this inspection.");
|
|
c.add("actual-diff-summary-with-file-references", /header\.html/.test(o.reply) && /Launch later/i.test(o.reply) && /Launch sooner/i.test(o.reply)
|
|
// The changed files and values are the setup's own literals, so the reply must report them. That the work
|
|
// is uncommitted is proven by the unchanged HEAD/status above; the reply need not use the word, only
|
|
// make no commit/push claim. The person asked about the website files, so the reply may leave out the
|
|
// unrelated existing-notes.txt change; its preservation is graded byte-for-byte above.
|
|
&& !/(?:I(?:'ve| have)? |successfully )(?:pushed|committed|published|deployed)|remote.*(?:has been|was) updated/i.test(o.reply),
|
|
"The answer describes the observed header diff and makes no commit, push or remote-update claim.");
|
|
c.add("real-native-inspection-only", o.runs.length === 2 && o.runs.every(run => run.status === "succeeded")
|
|
&& o.runs.filter(run => run.agentId === o.leadId).length === 1,
|
|
"The real prior workspace setup and one native inspection are the only provider executions.");
|
|
return c.result();
|
|
}
|
|
export const runtimeGitCases = [runtimeDefinition("workspaces.inspect-git", async ctx => {
|
|
if (ctx.input.execution.environment.id !== "local") throw new Error("Owned repository fixture requires the actual local runtime");
|
|
const state = await savedTaskWorkspace(ctx); const cwd = state.workspaceBefore.workspace.cwd;
|
|
await ownedGit(cwd, ["init", "--initial-branch=eval-fixture"]);
|
|
if ((await ownedGit(cwd, ["remote"])).trim()) throw new Error("Disposable synthetic task repository unexpectedly has a remote");
|
|
const header = path.join(cwd, "header.html"), notes = path.join(cwd, "existing-notes.txt");
|
|
await writeFile(header, `<h1>Launch later</h1>\n<!-- ${ctx.input.nonce} -->\n`, "utf8");
|
|
await writeFile(notes, "Existing unrelated notes before the user's edit.\n", "utf8");
|
|
await ownedGit(cwd, ["add", "--", "header.html", "existing-notes.txt"]);
|
|
await ownedGit(cwd, ["-c", "core.hooksPath=/dev/null", "-c", "user.name=Paperclip Eval Fixture", "-c", "user.email=fixture@invalid.example", "commit", "-m", "Synthetic baseline for the owned diff fixture"]);
|
|
await writeFile(header, `<h1>Launch sooner</h1>\n<!-- ${ctx.input.nonce} -->\n`, "utf8");
|
|
await writeFile(notes, `Existing unrelated notes preserved with human amendment ${ctx.input.nonce}.\n`, "utf8");
|
|
const lead = await ctx.input.api.get<RuntimeAgent>(`/api/agents/${ctx.input.fixtures.agent.id}`);
|
|
await ctx.input.api.patch(`/api/agents/${lead.id}`, { adapterConfig: { ...lead.adapterConfig, cwd } });
|
|
await ctx.input.api.patch("/api/instance/settings/experimental", { enableExperimentalFileViewer: true });
|
|
return { ...state, nativeInspectionCwd: cwd, noRemote: true, headBefore: await ownedGit(cwd, ["rev-parse", "HEAD"]),
|
|
diffBefore: await ownedGit(cwd, ["diff", "--", "header.html", "existing-notes.txt"]), statusBefore: await ownedGit(cwd, ["status", "--porcelain"]),
|
|
headerBefore: await readFile(header, "utf8"), notesBefore: await readFile(notes, "utf8") };
|
|
}, gradeInspectGit, {
|
|
sourceFiles: ["chat-catalog-domains/runtime-git.ts", "chat-catalog-domains/runtime-workspaces.ts", "chat-catalog-domains/runtime-support.ts", "chat-scenarios/workspaces.yaml"],
|
|
oracleVersion: "paperclip.chat-catalog.inspect-git.structural.v4", expectedRunCount: 2, attemptTimeoutMs: 540_000,
|
|
async observe(ctx, state) {
|
|
const cwd = String(state.nativeInspectionCwd);
|
|
state.workspaceAfter = await ctx.input.api.get(`/api/issues/${(state.task as Entity).id}/workspace`);
|
|
state.headAfter = await ownedGit(cwd, ["rev-parse", "HEAD"]); state.diffAfter = await ownedGit(cwd, ["diff", "--", "header.html", "existing-notes.txt"]);
|
|
state.statusAfter = await ownedGit(cwd, ["status", "--porcelain"]);
|
|
state.headerAfter = await readFile(path.join(cwd, "header.html"), "utf8"); state.notesAfter = await readFile(path.join(cwd, "existing-notes.txt"), "utf8");
|
|
return observeRuntime(ctx, state);
|
|
},
|
|
async verifyResult(ctx, state) {
|
|
const source = ctx.input.page.url(), task = state.task as Entity;
|
|
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier ?? task.id}`, { waitUntil: "domcontentloaded" });
|
|
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
|
|
await expect(ctx.input.page.getByRole("heading", { name: task.title!, exact: true }).first()).toBeVisible();
|
|
// The task side panel can start closed; open it through the control the page renders ("Show properties"
|
|
// in the task header), as a person would, then choose the Files tab from the panel's tab launcher.
|
|
const page = ctx.input.page;
|
|
const newTab = page.getByRole("button", { name: "Open a new tab", exact: true }).filter({ visible: true });
|
|
const toggle = page.getByRole("button", { name: /^(Show properties|Toggle side panel)$/ }).filter({ visible: true }).first();
|
|
const files = page.getByRole("option", { name: /^Files/ }).filter({ visible: true }).first();
|
|
await expect.poll(async () => await newTab.isVisible() || await files.isVisible() || await toggle.isVisible()).toBe(true);
|
|
if (!await newTab.isVisible() && !await files.isVisible()) await toggle.click();
|
|
await expect.poll(async () => await files.isVisible() || await newTab.isVisible()).toBe(true);
|
|
if (!await files.isVisible()) await newTab.click();
|
|
await files.click();
|
|
await expect(ctx.input.page.getByText("header.html", { exact: true }).first()).toBeVisible();
|
|
await expect(ctx.input.page.getByText("existing-notes.txt", { exact: true }).first()).toBeVisible();
|
|
await ctx.input.evidence("runtime-authorized-git-files-ui.json", { taskId: task.id, workspaceId: (state.workspaceBefore as { workspace: { id: string } }).workspace.id, url: ctx.input.page.url() });
|
|
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
|
|
},
|
|
})];
|