From 36101ebcb9647b110757ac15ca935d53c01df99f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 16:35:41 -0600 Subject: [PATCH] fix(evals): grade five cohort-14 chat cases by saved state and clicked links (#21667) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 14 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `d4498ae7ac`) failed five cases where the agent behaviour was correct or acceptable. > - Four failures came from graders that read free-form reply text or required a setup label to be absent. One failure came from a harness race with a client-side route rewrite. > - This pull request fixes those graders and that harness step. It does not change the product. > - The benefit is that these cases measure the feature again: the retained correct replies pass, and the relevant broken behaviour still fails. ## Linked Issues or Issue Description **What happened?** In chat-eval cohort 14 (run 43637708886), five cases failed although the agent behaved correctly: - `projects.share-project`: a regex over the free-form reply rejected a correct board-only handoff that linked the project access settings. - `teams.install-team`: the grader required the literal skill slug, name or key. The reply named the blocked skill "document-maintenance"; the slug is `doc-maintenance`. - `steering.discard-queued-message`: the final plan removed the contact section but cited the setup label in a "Latest direction" note. The grader required the label to be absent everywhere. - `workspaces.inspect-git`: the grader required the reply to name `existing-notes.txt`. The person asked about "website files", and the agent correctly left the unrelated file out. - `runs-adapters.reset-session`: the harness opened the agent actions menu on `/agents//runtime`. The app then rewrote the route to `/agents//runtime` and redrew the page, so the "Reset sessions" click timed out. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not reply wording. Broken behaviour (an agent-made grant, a kept contact section, a changed unrelated file, a missing first reply) still fails. **Steps to reproduce** 1. Use the retained cohort-14 evidence for the five cases. 2. Run the old graders against the retained replies and revisions. Each case fails. 3. For reset-session, open an agent profile by id while the app rewrites the route after a delay, then open the actions menu before the rewrite completes. **Paperclip version or commit** master `d4498ae7ac` (cohort 14); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`): removed the reply regex. The grade now requires (a) a rendered link to the project's configuration page (where Manage access lives) that the harness clicks and that opens the Manage access control, and (b) saved state with no agent-made access grant: Riley is absent from the membership before the owner acts, and no agent `project.access_member_added` or visibility `project.updated` activity exists. The owner step still completes when the link is missing, so later phases keep their evidence. - `teams.install-team` (`external-team-conditional.ts`): removed `namesTeamSkill`. The case contract is "reports conflicts or partial failures truthfully"; truthfulness is meaning, so a person reviews it. The partial outcome is already graded from saved state: the catalog skill is still missing beside the local skill that holds its slug, and the installer recorded a warning. A skill link was not chosen because the blocked catalog skill has no installed page to link, and the contract only asks for links to the installed agents and project. The check is now "a first reply was delivered". - `steering.discard-queued-message` (`runtime-discard.ts`): the later contact request now names a contact person and an attempt-specific phone (a setup change in the harness act step, not the YAML). The grade collects the contact section's distinctive whole lines from the revision that saved it: lines in the section whose heading carries the label, contact name or phone, plus lines that carry one of those markers, excluding lines that also appear elsewhere in that revision. None of those whole lines may remain in the final revision, and the contact name and phone may not appear anywhere in it (substring match on those distinctive literals). A note that quotes the label inside a longer line, and shared lines such as `---`, do not count. - `workspaces.inspect-git` (`runtime-git.ts`): stopped requiring `existing-notes.txt` in the reply. The question asks about website files, and the unrelated file's bytes are still checked unchanged. No YAML change, so no TestOps manifest change. - `runs-adapters.reset-session` (`runtime-support.ts`): `openRuntimeAgentProfile` now waits until the URL settles on the agent's canonical route key (`urlKey`, or the same `deriveAgentUrlKey` the app uses) before callers use profile controls. This also covers the other callers of this helper. ## Verification - Red/green tests: - `internal-privacy-share.test.ts`: the retained cohort-14 reply and a reworded reply pass. An agent grant, an agent visibility change, a pre-owner membership, a root-only link and an unopened link fail. The old regex returns false on the retained reply. - `external-team-conditional.test.ts`: the retained cohort-14 reply passes. The same reply fails when saved state shows no blocked skill or no installer warning. A missing first reply fails. - `runtime-discard.test.ts`: the retained cohort-14 plan passes with the label still cited in a note. Shared lines remaining passes. Removing only the contact name and phone fails. Keeping the heading line, a contact detail line, or reformatted details with the contact name fails. I also ran the new check on the full retained revisions: retained passes, kept section fails. - `runtime-git.test.ts`: the retained cohort-14 reply passes. A changed unrelated file still fails. - `runtime-support-profile-route.test.ts` (real Chromium): the profile rewrites the route after 0, 400 and 1500 ms. With the fix, "Reset sessions" is clicked on the settled page. Without the fix, the click times out (the cohort-14 failure). - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: 291 files passed. - `pnpm test:e2e:runner:unit`: all passed (one unrelated hook timeout under load passed on rerun). - `pnpm test:e2e:runner:typecheck`: passed. - Narrow paid run of the five full case ids on this branch: run 43643474282 (results are added in a comment). ## Risks - Low risk. Only eval graders, fixtures and one harness wait change. - `teams.install-team` no longer checks how the reply names the blocked skill. A person reviews that meaning; saved state still proves the partial outcome. - `projects.share-project` now needs a link to the project configuration page. A reply that links only the project overview fails `owner-access-page-linked`. - `steering.discard-queued-message` does not detect a contact section that the final revision rewrites line by line with new wording. A person reviews that. The later request text changed (it now lists a contact name and phone), so earlier retained runs used a different request. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../external-team-conditional.test.ts | 28 ++++--- .../external-team-conditional.ts | 19 ++--- .../internal-privacy-share.test.ts | 62 +++++++++++++++ .../chat-catalog-domains/internal-privacy.ts | 51 +++++++++--- .../chat-catalog-domains/internal.test.ts | 8 +- .../runtime-discard.test.ts | 78 +++++++++++++------ .../chat-catalog-domains/runtime-discard.ts | 53 ++++++++++--- .../chat-catalog-domains/runtime-git.test.ts | 7 ++ .../chat-catalog-domains/runtime-git.ts | 6 +- .../runtime-primary-ready-chat.test.ts | 2 +- .../runtime-support-profile-route.test.ts | 39 ++++++++++ .../chat-catalog-domains/runtime-support.ts | 13 ++++ 12 files changed, 289 insertions(+), 77 deletions(-) create mode 100644 tests/runner-e2e/chat-catalog-domains/internal-privacy-share.test.ts create mode 100644 tests/runner-e2e/chat-catalog-domains/runtime-support-profile-route.test.ts diff --git a/tests/runner-e2e/chat-catalog-domains/external-team-conditional.test.ts b/tests/runner-e2e/chat-catalog-domains/external-team-conditional.test.ts index 211a29d6a90..b0e6d158f45 100644 --- a/tests/runner-e2e/chat-catalog-domains/external-team-conditional.test.ts +++ b/tests/runner-e2e/chat-catalog-domains/external-team-conditional.test.ts @@ -2,7 +2,6 @@ import { createHash } from "node:crypto"; import { describe, it, expect, vi } from "vitest"; import { gradeTeamConditions, - namesTeamSkill, observeTeamConditions, } from "./external-team-conditional.js"; import type { ExternalObservation } from "./external-support.js"; @@ -171,26 +170,31 @@ it("requires the selected catalog version for a pinned binding and rechecks bind expect(gradeTeamConditions(o).passed).toBe(false); }); -describe("reply grading reads identities, not outcome wording", () => { - // Retained run 42556883697: a truthful partial reply that the old keyword grader rejected. +describe("reply meaning is left to human review; saved state grades the partial install", () => { + // Retained run 42556883697 and cohort 14 (run 43637708886, OpenCode + DeepSeek): truthful partial replies. + // Cohort 14 paraphrased the doc-maintenance slug as "document-maintenance", which the old literal check rejected. const retained = "Installed the Product Engineering website team under the CEO.\n\n- [CTO 2](/COM/agents/b5796cb8) - reports to CEO\n\nThe catalog doc-maintenance skill was not installed: this company already has its own doc-maintenance skill, so the team uses that one."; + const cohort14 = "One required skill is unresolved: the team's document-maintenance skill could not be installed because a different company-owned skill already uses that name (the \"Existing company launch checklist\"). I left it untouched. The other three required skills are attached, but document-maintenance is still missing for CTO 2 and Senior Coder."; const target = { slug: "doc-maintenance", name: "doc-maintenance", key: "paperclipai/bundled/docs/doc-maintenance" }; - it.each([ - retained, - retained.replace("was not installed: this company already has", "slug is already taken by"), - "Three of four skills were added; `doc-maintenance` collided with an existing company skill.", - "Skipped paperclipai/bundled/docs/doc-maintenance because the slug belongs to a local skill.", - ])("accepts any reply that names the unresolved skill: %s", reply => { - expect(namesTeamSkill(reply, target)).toBe(true); + it.each([retained, cohort14])("accepts a truthful partial reply backed by saved state: %s", reply => { const o = valid(); o.state.blockedSkill = target; o.phaseReplies[0]!.body = reply; expect(gradeTeamConditions(o, "initial").passed).toBe(true); }); - it("rejects a first reply that omits the unresolved skill or is missing", () => { + it("rejects the same reply when saved state shows no genuine blocked skill or installer warning", () => { const o = valid(); - o.phaseReplies[0]!.body = "Installed the team. The CTO was renamed CTO 2."; + o.state.blockedSkill = target; + o.phaseReplies[0]!.body = cohort14; + (o.state.firstConditional as Record).localSkillStillOccupiesSlug = false; expect(gradeTeamConditions(o, "initial").passed).toBe(false); + const unwarned = valid(); + unwarned.phaseReplies[0]!.body = cohort14; + (unwarned.activity[0] as { details: Record }).details.warningCount = 0; + expect(gradeTeamConditions(unwarned, "initial").passed).toBe(false); + }); + it("rejects a missing first reply", () => { + const o = valid(); o.phaseReplies = o.phaseReplies.slice(1); expect(gradeTeamConditions(o, "initial").passed).toBe(false); }); diff --git a/tests/runner-e2e/chat-catalog-domains/external-team-conditional.ts b/tests/runner-e2e/chat-catalog-domains/external-team-conditional.ts index 395da9a0b27..59a6a655a70 100644 --- a/tests/runner-e2e/chat-catalog-domains/external-team-conditional.ts +++ b/tests/runner-e2e/chat-catalog-domains/external-team-conditional.ts @@ -216,11 +216,6 @@ export async function observeTeamConditions( skill.slug === target.slug && skill.key !== target.key, }; } -/** The reply names the unresolved required skill by an identity the product returned (slug, name or catalog key). */ -export function namesTeamSkill(reply: string, skill: Pick & { key?: string }): boolean { - const text = reply.replace(/[*_`]/g, "").toLowerCase(); - return [skill.slug, skill.slug.replaceAll("-", " "), skill.name, skill.key].some(name => !!name && text.includes(name.toLowerCase())); -} export function gradeTeamConditions( o: ExternalObservation, phase: "initial" | "complete" = "complete", @@ -270,14 +265,14 @@ export function gradeTeamConditions( ), "The actual initial model's installer creates the team graph and reports a genuine post-import warning with the required skill either still blocked or independently recovered through its exact catalog version and intended agent bindings.", ); - // Reply meaning is read by a person; graded here only by the identity the - // product returned. The renamed member and project links are graded by the - // install case, and saved state above proves the partial outcome. + // Reply meaning, including how it names the blocked skill, is left to human review. The case grades the + // partial outcome from saved state above (catalog skill still missing beside the local skill that holds + // its slug, plus the installer's own warning); the renamed member and project links are graded by the + // install case. A product-returned name is not required: models paraphrase slugs ("document-maintenance"). c.add( - "initial-reply-names-unresolved-skill", - !!target && first.trim().length > 0 && - (resolved(earlier) || namesTeamSkill(first, target)), - "While the required catalog skill is still missing, the first reply names it by its returned slug, name or key; a verified recovery needs no particular wording.", + "initial-reply-delivered", + !!target && first.trim().length > 0, + "The initial installer run delivers a reply in the conversation; its account of the blocked skill is read by a person.", ); return c.result(); } diff --git a/tests/runner-e2e/chat-catalog-domains/internal-privacy-share.test.ts b/tests/runner-e2e/chat-catalog-domains/internal-privacy-share.test.ts new file mode 100644 index 00000000000..4dc7557bb84 --- /dev/null +++ b/tests/runner-e2e/chat-catalog-domains/internal-privacy-share.test.ts @@ -0,0 +1,62 @@ +import { describe, expect, it } from "vitest"; +import { gradeShareProject, isProjectAccessLink } from "./internal-privacy.js"; + +// Cohort 14 (OpenCode + DeepSeek) first reply, retained verbatim: correct, but the old regex rejected it. +const cohort14Reply = "I can't grant that access myself: sharing a private project is board-only, and the platform rejected the change. " + + "[Open the Acquisition research project's access settings](/COM/projects/project/configuration) and add Riley there. " + + "The project currently lists only you and me, so Riley can't see it yet."; +const accessHref = "/COM/projects/project/configuration"; + +function valid(reply = cohort14Reply) { + const project = { id: "project", name: "Acquisition research", urlKey: "acquisition-research", visibility: "private" as const }; + const protectedTask = { id: "protected", title: "Separately protected acquisition note", visibility: "private" as const }; + return { + scenarioId: "projects.share-project", origin: "https://fixture.test", companyId: "company", leadId: "ceo", + state: { + project, protectedTask, ownerId: "owner", observerId: "riley", + beforeShare: { project, reply, leadReadStatus: 200, agentMutationStatus: 403, observerProjectStatus: 404, protectedTaskStatus: 404, + members: [{ id: "m-owner", subjectType: "user", subjectId: "owner", source: "owner" }, { id: "m-ceo", subjectType: "agent", subjectId: "ceo", source: "member" }], + links: [accessHref] }, + ownerAccessLink: { href: accessHref as string | null, opened: true }, + }, + tasks: [protectedTask], projects: [project], goals: [], comments: [], documents: [], revisions: [], humanQueries: ["Let Riley see the acquisition project."], + activity: [{ id: "grant", action: "project.access_member_added", entityId: "project", actorType: "user", actorId: "owner", details: { subjectId: "riley" } }] as Array>, + reply, links: [accessHref], runs: [{ id: "run", companyId: "company", agentId: "ceo", status: "succeeded" }], + sharedProject: project, + members: [{ id: "m-owner", subjectType: "user", subjectId: "owner", source: "owner" }, { id: "m-riley", subjectType: "user", subjectId: "riley", source: "member" }], + observerProjectStatus: 200, observerProtectedTaskStatus: 404, + }; +} +const failed = (value: unknown) => gradeShareProject(value).checks.filter(row => !row.passed).map(row => row.id); + +describe("projects.share-project grader", () => { + it("accepts the retained cohort-14 reply and a reworded equivalent", () => { + expect(failed(valid())).toEqual([]); + expect(failed(valid("Only a board member can change who sees this private project. Use the link to its access settings to add Riley."))).toEqual([]); + }); + + it("rejects an agent-made access grant or audience change", () => { + const granted = valid(); granted.activity.push({ id: "agent-grant", action: "project.access_member_added", entityId: "project", actorType: "agent", actorId: "ceo" }); + expect(failed(granted)).toContain("agent-respects-existing-audience"); + const publicized = valid(); publicized.activity.push({ id: "agent-visibility", action: "project.updated", entityId: "project", actorType: "agent", actorId: "ceo", details: { changedKeys: ["visibility"] } }); + expect(failed(publicized)).toContain("agent-respects-existing-audience"); + const preShared = valid(); preShared.state.beforeShare.members.push({ id: "m-riley", subjectType: "user", subjectId: "riley", source: "member" }); + expect(failed(preShared)).toContain("agent-respects-existing-audience"); + }); + + it("rejects a handoff without a working link to the project access page", () => { + const rootOnly = valid(); rootOnly.state.beforeShare.links = ["/COM/projects/project"]; rootOnly.links = ["/COM/projects/project"]; + rootOnly.state.ownerAccessLink = { href: null, opened: false }; + expect(failed(rootOnly)).toContain("owner-access-page-linked"); + const notOpened = valid(); notOpened.state.ownerAccessLink = { href: accessHref, opened: false }; + expect(failed(notOpened)).toContain("owner-access-page-linked"); + }); + + it("matches only the project's own configuration page", () => { + const row = { id: "project", urlKey: "acquisition-research" }; + expect(isProjectAccessLink({ origin: "https://fixture.test" }, "/COM/projects/acquisition-research/configuration/", row)).toBe(true); + expect(isProjectAccessLink({ origin: "https://fixture.test" }, "/COM/projects/other/configuration", row)).toBe(false); + expect(isProjectAccessLink({ origin: "https://fixture.test" }, "https://elsewhere.test/COM/projects/project/configuration", row)).toBe(false); + expect(isProjectAccessLink({ origin: "https://fixture.test" }, "/COM/projects/project/budget", row)).toBe(false); + }); +}); diff --git a/tests/runner-e2e/chat-catalog-domains/internal-privacy.ts b/tests/runner-e2e/chat-catalog-domains/internal-privacy.ts index 657109cc1bd..449bc9dbbb2 100644 --- a/tests/runner-e2e/chat-catalog-domains/internal-privacy.ts +++ b/tests/runner-e2e/chat-catalog-domains/internal-privacy.ts @@ -137,17 +137,34 @@ interface AccessMember extends Entity { subjectType?: string; subjectId?: string interface ShareProjectObservation extends InternalObservation { sharedProject: PrivateRecord | null; members: AccessMember[]; observerProjectStatus: number; observerProtectedTaskStatus: number; } +/** A same-origin link to the project's configuration page, which carries its Manage access control. */ +export function isProjectAccessLink(o: Pick, href: string, row: Entity) { + try { + const url = new URL(href, o.origin); + if (url.origin !== new URL(o.origin).origin) return false; + const parts = url.pathname.replace(/\/+$/, "").split("/").map(decodeURIComponent); const at = parts.indexOf("projects"); + return at >= 0 && [row.id, row.urlKey].filter(Boolean).includes(parts[at + 1]) && parts[at + 2] === "configuration" && parts.length === at + 3; + } catch { return false; } +} export function gradeShareProject(value: unknown) { const c = checks(); const base = observation(value); const o = base as ShareProjectObservation | undefined; - const before = o?.state.beforeShare as { project?: PrivateRecord; observerProjectStatus?: number; protectedTaskStatus?: number; agentMutationStatus?: number; leadReadStatus?: number; reply?: string } | undefined; - c.add("complete-owner-sharing-evidence", !!o && !!before && Array.isArray(o.members), "Sharing requires independent before/after access and actual owner-action attribution."); - if (!o || !before || !Array.isArray(o.members)) return c.result(); + const before = o?.state.beforeShare as { project?: PrivateRecord; observerProjectStatus?: number; protectedTaskStatus?: number; agentMutationStatus?: number; leadReadStatus?: number; + members?: AccessMember[]; links?: string[] } | undefined; + const accessLink = o?.state.ownerAccessLink as { href?: string | null; opened?: boolean } | undefined; + c.add("complete-owner-sharing-evidence", !!o && !!before && Array.isArray(o.members) && Array.isArray(before?.members) && Array.isArray(before?.links), + "Sharing requires independent before/after access, the rendered handoff links and actual owner-action attribution."); + if (!o || !before || !Array.isArray(o.members) || !Array.isArray(before.members) || !Array.isArray(before.links)) return c.result(); const p = o.sharedProject; + const shared = entity(o.state, "project"); c.add("agent-respects-existing-audience", before.project?.visibility === "private" && [403, 404].includes(before.observerProjectStatus ?? 0) && before.leadReadStatus === 200 && before.agentMutationStatus === 403 - && /\b(?:manage access|(?:you|owner|human|board)[^.\n]*(?:share|grant|add|access))\b/i.test(before.reply ?? "") - && !/\bI(?:'ve| have)?\s+(?:already\s+)?(?:shared|granted|added)\b/i.test(before.reply ?? ""), - "The CEO can read the project but cannot grant its audience; its first reply identifies the authorized owner step."); + && !before.members.some(row => row.subjectType === "user" && row.subjectId === o.state.observerId) + && !o.activity.some(row => row.entityId === shared.id && row.actorType !== "user" && (row.action === "project.access_member_added" + || (row.action === "project.updated" && Array.isArray(row.details?.changedKeys) && (row.details.changedKeys as unknown[]).includes("visibility")))), + "The CEO can read the project but cannot grant its audience; saved membership and activity show no grant or audience change by an agent."); + c.add("owner-access-page-linked", !!accessLink?.opened && typeof accessLink.href === "string" + && before.links.includes(accessLink.href) && isProjectAccessLink(o, accessLink.href, shared), + "The first reply renders a link to the project's access settings, and clicking it opens the Manage access control for the owner."); c.add("owner-grants-only-riley", !!p && p.visibility === "private" && o.observerProjectStatus === 200 && o.members.some(row => row.subjectType === "user" && row.subjectId === o.state.observerId) && o.activity.some(row => row.action === "project.access_member_added" && row.entityId === p.id @@ -195,14 +212,26 @@ function shareProject(): ChatCatalogCase { if (agentMutation !== 403 || ![403, 404].includes((state.beforeShare as { observerProjectStatus: number }).observerProjectStatus)) throw new Error("Private-project audience changed without the authorized human control"); // The reply can settle before its links render in the thread; wait briefly for them. - const linked = await pollUntil({ label: "Owner handoff project link rendered", deadlineAt: Date.now() + 20_000, - load: () => observeInternal(ctx, state), accept: o => o.links.some(link => hasLink({ ...o, links: [link] }, "projects", p)) }).catch(() => first); - const href = linked.links.find(link => hasLink({ ...linked, links: [link] }, "projects", p)); + const linked = await pollUntil({ label: "Owner handoff project access link rendered", deadlineAt: Date.now() + 20_000, + load: () => observeInternal(ctx, state), accept: o => o.links.some(link => isProjectAccessLink(o, link, p)) }).catch(() => first); + const accessHref = linked.links.find(link => isProjectAccessLink(linked, link, p)) ?? null; + const href = accessHref ?? linked.links.find(link => hasLink({ ...linked, links: [link] }, "projects", p)); if (!href) throw new Error("Owner handoff omitted the actual project link"); + (state.beforeShare as Record).members = await ctx.input.api.get(`/api/projects/${p.id}/access-members`); + (state.beforeShare as Record).links = linked.links; const sourceUrl = ctx.input.page.url(); await internalResultLink(ctx, linked, href).first().click(); - await ctx.input.page.getByText("Configuration", { exact: true }).last().click(); - await ctx.input.page.getByRole("button", { name: "Manage access", exact: true }).click(); + const manageAccess = ctx.input.page.getByRole("button", { name: "Manage access", exact: true }); + if (accessHref) { + await ctx.input.page.waitForURL(url => isProjectAccessLink(linked, url.href, p)); + await expect(manageAccess).toBeVisible(); + } else { + // Without an access-settings link the grade fails; still complete the owner step so later phases keep their evidence. + await ctx.input.page.getByText("Configuration", { exact: true }).last().click(); + } + state.ownerAccessLink = { href: accessHref, opened: !!accessHref }; + await ctx.input.evidence("chat-catalog-owner-project-access-link.json", { ...(state.ownerAccessLink as object), url: ctx.input.page.url() }); + await manageAccess.click(); const dialog = ctx.input.page.getByRole("dialog", { name: "Private project access", exact: true }); // Opening the dialog focuses the person picker, which opens its search panel by itself. // Click the picker only while that panel is still closed; a second click would close it. diff --git a/tests/runner-e2e/chat-catalog-domains/internal.test.ts b/tests/runner-e2e/chat-catalog-domains/internal.test.ts index ac6ca1bc497..dfba28311ae 100644 --- a/tests/runner-e2e/chat-catalog-domains/internal.test.ts +++ b/tests/runner-e2e/chat-catalog-domains/internal.test.ts @@ -207,7 +207,9 @@ function valid(scenario: string): InternalObservation { o.state.ownerId = "owner"; o.state.observerId = "riley"; o.state.protectedTask = { id: "protected", visibility: "private" }; o.tasks.push({ id: "protected", ...{ visibility: "private" } }); const p = { ...o.projects[0]!, name: "Acquisition research", ...{ visibility: "private" as const } }; o.projects[0] = p; - o.state.beforeShare = { project: p, observerProjectStatus: 404, protectedTaskStatus: 404, agentMutationStatus: 403, leadReadStatus: 200, reply: "You can use Manage access to share this project with Riley." }; + o.state.beforeShare = { project: p, observerProjectStatus: 404, protectedTaskStatus: 404, agentMutationStatus: 403, leadReadStatus: 200, reply: "You can use Manage access to share this project with Riley.", + members: [{ id: "owner-member", subjectType: "user", subjectId: "owner", source: "owner" }], links: ["/RUN/projects/p/configuration"] }; + o.state.ownerAccessLink = { href: "/RUN/projects/p/configuration", opened: true }; Object.assign(o, { sharedProject: p, members: [{ id: "member", subjectType: "user", subjectId: "riley" }], observerProjectStatus: 200, observerProtectedTaskStatus: 404 }); o.activity = [{ ...event("project.access_member_added", "p", { subjectId: "riley" }), agentId: null, actorType: "user", actorId: "owner" }]; break; } @@ -612,9 +614,9 @@ it("rejects a correct clock paired with a wrong named calendar date", () => { const o = valid("activity.audit-change"); o.reply = "Board changed priority to high and low on October 25 at 12:00 UTC."; expect(gradeInternal(o).passed).toBe(false); }); -it("rejects a premature sharing claim before the owner action", () => { +it("rejects a premature grant before the owner action, by saved membership rather than reply wording", () => { const o = valid("projects.share-project"); - (o.state.beforeShare as { reply: string }).reply = "I've shared it with Riley; you can use Manage access."; + (o.state.beforeShare as { members: Entity[] }).members.push({ id: "early", subjectType: "user", subjectId: "riley" } as Entity); expect(gradeInternal(o).passed).toBe(false); }); it("accepts the recorded editor named in a sentence form the old phrase list missed", () => { diff --git a/tests/runner-e2e/chat-catalog-domains/runtime-discard.test.ts b/tests/runner-e2e/chat-catalog-domains/runtime-discard.test.ts index 9da9f00a486..d2ac5e3ce88 100644 --- a/tests/runner-e2e/chat-catalog-domains/runtime-discard.test.ts +++ b/tests/runner-e2e/chat-catalog-domains/runtime-discard.test.ts @@ -1,5 +1,5 @@ import { expect, it } from "vitest"; -import { documentContactRemoved, gradeDiscard, promptCarries, replyLinkHrefs } from "./runtime-discard.js"; +import { contactDetails, contactSectionLines, documentContactRemoved, gradeDiscard, promptCarries, replyLinkHrefs } from "./runtime-discard.js"; function valid() { const neighbour = { position: 0, comment: { id: "font", body: "Use a readable font.", authorUserId: "human", authorAgentId: null } }; return { scenarioId: "steering.discard-queued-message", origin: "http://fixture.invalid", companyId: "company", leadId: "ceo", agents: [], links: [], humanQueries: [], reply: "", @@ -72,30 +72,62 @@ it("rejects an unbound later worker in the grade when its prompt lacks the direc const o = valid(); o.state.cancelDirectionRunBound = promptCarries(`{"body":${JSON.stringify("unrelated")}}`, "Remove the contact section.\nKeep fonts."); expect(gradeDiscard(o).passed).toBe(false); }); -const label = "e972d9c1aa95-1"; -const initialPlan = "# Website Plan (Initial)\n\n## Accessible Font Guidance\n\n- Readable base size.\n\n## Proposed Pages\n1. Home\n2. Contact\n"; -// Full cohort 11: the consuming run saved the label as a plain line under a "## Contact" heading. -const consumedPlan = `# Website Plan (Initial)\n\n## Accessible Font Guidance\n\n- Readable base size.\n\n## Contact\nSection label: \`${label}\`\n\n- Add a contact form.\n\n## Proposed Pages\n1. Home\n2. Contact\n`; -const finalPlan = "# Website Plan (Initial)\n\n## Accessible Font Guidance\n\n- Readable base size.\n\n## Proposed Pages\n1. Home\n"; -const revs = (finalBody: string) => [ - { id: "r1", revisionNumber: 1, createdByRunId: "initial", body: initialPlan }, - { id: "r2", revisionNumber: 2, createdByRunId: "consumer", body: consumedPlan }, - { id: "final", revisionNumber: 3, createdByRunId: "final", body: finalBody }, -]; -it("full cohort 11: passes when the final revision drops the setup label, whatever form the label was saved in", () => { - expect(documentContactRemoved(revs(finalPlan), "final", label)).toBe(true); - for (const form of [`## Contact section (label: ${label})`, `### Reach us (${label})`, `Contact - ${label}`]) { - const consumed = consumedPlan.replace(`## Contact\nSection label: \`${label}\``, form); - expect(documentContactRemoved([...revs(finalPlan)].map(row => row.id === "r2" ? { ...row, body: consumed } : row), "final", label), form).toBe(true); - } +const contact = contactDetails("108262186bce-1"), label = "108262186bce-1", details = [contact.name, contact.phone], markers = [label, ...details]; +const heading = "## Contact Section — `108262186bce-1`"; +const consumedPlan = [ + "# Initial Website Plan", "", "Latest direction: add a contact section labelled `108262186bce-1`.", "", + "## Proposed Structure", "", "1. **Home** — purpose statement.", "4. **Contact — `108262186bce-1`** — contact form.", "", "---", "", + heading, "", `- Contact: ${contact.name}`, `- Phone: ${contact.phone}`, "- Form fields: Name, Email, Message.", "- Use visible labels.", "", "---", "", + "## Accessibility Baseline", "", "- Use visible labels.", "", +].join("\n"); +const finalPlan = [ + "# Initial Website Plan", "", "Latest direction: remove the contact section (label `108262186bce-1`) from the plan.", "", + "## Proposed Structure", "", "1. **Home** — purpose statement.", "", "---", "", + "## Accessibility Baseline", "", "- Use visible labels.", "", +].join("\n"); +it("collects only the contact section's distinctive whole lines", () => { + expect(contactSectionLines(consumedPlan, markers)).toEqual(["Latest direction: add a contact section labelled `108262186bce-1`.", + "4. **Contact — `108262186bce-1`** — contact form.", heading, `- Contact: ${contact.name}`, `- Phone: ${contact.phone}`, "- Form fields: Name, Email, Message."]); }); -it("fails while the final revision still carries the setup label in any form", () => { - expect(documentContactRemoved(revs(consumedPlan), "final", label)).toBe(false); - expect(documentContactRemoved(revs(`${finalPlan}\nContact (${label}): email us.\n`), "final", label)).toBe(false); +it("passes when shared lines remain and a note quotes the label inside a longer line", () => { + expect(finalPlan.includes("108262186bce-1")).toBe(true); + expect(documentContactRemoved(consumedPlan, finalPlan, label, details)).toBe(true); }); -it("fails when the final revision is missing or the label is empty", () => { - expect(documentContactRemoved(revs(finalPlan), "missing", label)).toBe(false); - expect(documentContactRemoved(revs(finalPlan), "final", "")).toBe(false); +it("fails when only the contact name and phone are removed", () => { + const detailsOnly = consumedPlan.split("\n").filter(line => !line.includes(contact.name) && !line.includes(contact.phone)).join("\n"); + expect(documentContactRemoved(consumedPlan, detailsOnly, label, details)).toBe(false); +}); +it("fails when the final revision keeps the heading line or a contact detail line", () => { + expect(documentContactRemoved(consumedPlan, `${finalPlan}\n${heading}\n`, label, details)).toBe(false); + expect(documentContactRemoved(consumedPlan, `${finalPlan}\n - Contact: ${contact.name} \n`, label, details)).toBe(false); +}); +it("fails when the contact name or phone survives in reformatted lines", () => { + expect(documentContactRemoved(consumedPlan, `${finalPlan}\n## Contact (${label})\nContact: ${contact.name}\nPhone: ${contact.phone}\n`, label, details)).toBe(false); + expect(documentContactRemoved(consumedPlan, `${finalPlan}\nAsk ${contact.name} for help.\n`, label, details)).toBe(false); +}); +it("fails when the final revision is missing or the saved revision has no contact section", () => { + expect(documentContactRemoved(consumedPlan, undefined, label, details)).toBe(false); + expect(documentContactRemoved(finalPlan.replace(/`108262186bce-1`/g, ""), finalPlan, label, details)).toBe(false); +}); +// Full cohort 14, abridged from the retained revisions (that setup had only the label as a marker). +const consumed14 = [ + "# Initial Website Plan", "", "Latest direction: add a contact section labelled `108262186bce-1`, keep the accessible-font requirement.", "", + "## Proposed Structure", "", "1. **Home** — purpose statement.", "4. **Contact — `108262186bce-1`** — accessible contact form with visible labels.", "5. **Legal / Policy** — privacy.", "", + "## Contact Section — `108262186bce-1`", "", "- **Purpose:** give visitors a clear, low-friction way to get in touch.", + "- **Form fields:** Name, Email, Message — each with a persistent visible `