From e879a24293d751ff0d64161ffddccdede200e423 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 11 Nov 2026 23:44:15 -0600 Subject: [PATCH] test(evals): check the wrong-resolver reply links Alex's request and task-link labels match their targets (#21028) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades saved product state and rendered links, never reply wording > - Full chat-eval cohort 4 had two cases whose graders passed replies with a structural flaw that the grader could see but did not check > - `connection-setup.wrong-person-resolves` promises that chat explains whose action is required, but its grader had no reply check at all > - `navigation-preferences.find-project-view` passed a reply with a link labelled COM-2 that opened COM-1 > - This pull request adds two structural reply checks, with calibration tests for valid, reworded-valid and broken replies > - The benefit is that a reply that does not point to the request, or that sends a person to the wrong task, fails the eval ## Linked Issues or Issue Description **What happened?** - `connection-setup.wrong-person-resolves`: another member said "I will finish Alex's connection request for them". The product refused that member (403 on setup options, phase, complete and decline) and kept Alex's card pending. The grader checked only that. The reply's content was not checked in any form. - `navigation-preferences.find-project-view`: the saved reply contained `[COM-2 Launch open website](/COM/issues/COM-1)`. The grader passed it. **Expected behavior** - The wrong-resolver reply points the person at Alex's actual pending request. The card on that page says "Waiting for Alex", which is whose action is required. The grader checks the link from saved state, and the browser step follows the reply's own link to that card. - Every task link in the find-project-view reply opens the task that its visible identifier names. **Steps to reproduce** 1. Run the two cases (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`) on master at f1aef0cb58. 2. Read the saved replies and the grader checks in the evidence. ## What Changed - `external-wrong-resolver.ts`: new check `reply-links-alex-pending-request`. The reply must render a same-origin link to the related task that holds Alex's card (`//issues/`). `verifyResult` now follows that rendered link (not a constructed URL) and confirms the "Waiting for Alex" card. Setup records the company prefix. Oracle `paperclip.chat-catalog.external.v2` -> `v3`. Reply wording is still not graded. - `chat-rendered-links.ts`: new shared helper `mislabeledIssueLinks(text, identifierFor?)`. It returns inline Markdown links whose label names a task identifier (same prefix as the target) other than the task the link opens. Labels without such a token, and non-task targets, are not judged. Task-id targets are compared through an optional resolver. - `identity-navigation-preferences.ts`: find-project-view adds `task-links-match-their-labels` with that helper. Oracle `paperclip.chat-catalog.identity.v1` -> `paperclip.chat-catalog.identity.find-project-view.v2`; `chat-rendered-links.ts` joins its source files. - `chat-catalog-descriptors.json`: the two descriptors carry the new oracle versions and source files. - The scenario YAML does not change, so the testops manifest has no `changedFields` entry. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.test.ts tests/runner-e2e/chat-rendered-links.test.ts tests/runner-e2e/chat-catalog-domains/identity.test.ts`: 92 tests pass. - Calibration, wrong resolver: valid (links CMP-1), reworded valid (links by task id, absolute same-origin link) pass; no link, another task, another company prefix, another origin, and a sub-path fail. Red/green: with the master grader, the "does not point at Alex's request" test fails. - Calibration, labels: valid, reworded (label without an identifier, document anchor, task-id target resolved), and a different-prefix token pass; the recorded cohort-4 link `[COM-2 Launch open website](/COM/issues/COM-1)` fails. The helper flags exactly that link in the retained cohort-4 reply. - Retained cohort-4 wrong-resolver reply links `/COM/issues/COM-1` (the related task), so it passes the new link check. Its wording ("completing it there is the right move") is a guidance problem, fixed in #21027, not something saved state can grade. - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-campaign chat-catalog`: 3856 tests pass. `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass. `tsc -p tests/runner-e2e/tsconfig.json` passes. - A paid run of both cases with the product fixes is linked in a comment below. ## Risks - A valid reply that names Alex but links nothing now fails wrong-person-resolves. The case contract says chat explains whose action is required; the card is where Paperclip shows it, so pointing to it is the structural form of that promise. - The label check could flag a label that intentionally names two tasks of the same company and links only one. That reply would still be misleading, so failing it is acceptable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- tests/runner-e2e/chat-catalog-descriptors.json | 5 +++-- .../external-wrong-resolver.test.ts | 9 ++++++--- .../external-wrong-resolver.ts | 11 ++++++++--- .../identity-navigation-preferences.ts | 6 +++++- .../chat-catalog-domains/identity.test.ts | 10 ++++++++++ tests/runner-e2e/chat-rendered-links.test.ts | 15 ++++++++++++++- tests/runner-e2e/chat-rendered-links.ts | 16 ++++++++++++++++ 7 files changed, 62 insertions(+), 10 deletions(-) diff --git a/tests/runner-e2e/chat-catalog-descriptors.json b/tests/runner-e2e/chat-catalog-descriptors.json index 1e3f693f842..baa4107cd09 100644 --- a/tests/runner-e2e/chat-catalog-descriptors.json +++ b/tests/runner-e2e/chat-catalog-descriptors.json @@ -2680,7 +2680,7 @@ "../../server/src/services/connection-intents.ts", "chat-scenarios/connection-setup.yaml" ], - "oracleVersion": "paperclip.chat-catalog.external.v2", + "oracleVersion": "paperclip.chat-catalog.external.v3", "expectedRunCount": 2, "expectedHumanMessageCount": 1, "minimumRunCount": 2, @@ -3670,9 +3670,10 @@ "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/identity-navigation-preferences.ts", + "chat-rendered-links.ts", "chat-scenarios/navigation-preferences.yaml" ], - "oracleVersion": "paperclip.chat-catalog.identity.v1", + "oracleVersion": "paperclip.chat-catalog.identity.find-project-view.v2", "expectedRunCount": 1, "turnTimeoutMs": 120000, "attemptTimeoutMs": 300000, diff --git a/tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.test.ts b/tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.test.ts index 7d1f552fc20..5c716d7531b 100644 --- a/tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.test.ts +++ b/tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.test.ts @@ -1,6 +1,6 @@ import {describe,it,expect} from "vitest"; import {gradeWrongResolver} from "./external-wrong-resolver.js"; -function valid(){const original={id:"card",kind:"connection_intent",status:"pending",addresseeUserId:"alex",sourceRunId:"alex-run",payload:{serviceSlug:"notion"}};return {scenarioId:"connection-setup.wrong-person-resolves",companyId:"company",leadId:"ceo",reply:"Only Alex can authorize their own Notion connection request. They need to finish that setup themselves.",phaseReplies:[],connections:[],profiles:[],secrets:[],endpoints:[],interactions:[],activity:[],links:[],humanQueries:[],runs:[{id:"alex-run",contextSnapshot:{issueId:"related"},agentId:"ceo",status:"succeeded",runtimeMode:"native"}],state:{original,after:{...original,result:null as unknown,resolvedByUserId:null as string|null,resolvedByAgentId:null as string|null},related:{id:"related"},alexId:"alex",otherId:"other",relatedVisible:true,otherSessionConfirmed:true,denied:{setupOptions:403,phase:403,complete:403,decline:403},waitingVisible:true,connectControlAbsent:true,relatedStatusUnchanged:true,alexSourceRunCount:1,canonicalSourceDifferent:true}};} +function valid(){const original={id:"card",kind:"connection_intent",status:"pending",addresseeUserId:"alex",sourceRunId:"alex-run",payload:{serviceSlug:"notion"}};return {scenarioId:"connection-setup.wrong-person-resolves",origin:"http://paperclip.test",companyId:"company",leadId:"ceo",reply:"Only Alex can authorize their own Notion connection request on [CMP-1](/CMP/issues/CMP-1). They need to finish that setup themselves.",phaseReplies:[],connections:[],profiles:[],secrets:[],endpoints:[],interactions:[],activity:[],links:["/CMP/issues/CMP-1"],humanQueries:[],runs:[{id:"alex-run",contextSnapshot:{issueId:"related"},agentId:"ceo",status:"succeeded",runtimeMode:"native"}],state:{original,after:{...original,result:null as unknown,resolvedByUserId:null as string|null,resolvedByAgentId:null as string|null},related:{id:"related",identifier:"CMP-1"},alexId:"alex",otherId:"other",relatedVisible:true,otherSessionConfirmed:true,denied:{setupOptions:403,phase:403,complete:403,decline:403},waitingVisible:true,connectControlAbsent:true,relatedStatusUnchanged:true,alexSourceRunCount:1,canonicalSourceDifferent:true,companyPrefix:"CMP"}};} describe("actual addressed connection resolver boundary",()=>{ it("accepts only a genuine pending Alex request with an independently denied different human",()=>expect(gradeWrongResolver(valid()).passed).toBe(true)); it("rejects impersonation, unauthorized completion, account creation and released waiting work",()=>{ @@ -17,8 +17,11 @@ describe("actual addressed connection resolver boundary",()=>{ (o:ReturnType)=>{o.state.waitingVisible=false;}, ]){const o=valid();mutate(o);expect(gradeWrongResolver(o).passed).toBe(false);} }); - it("grades the refused routes and pending card, not the reply's words",()=>{ - for(const reply of ["I cannot authorize Alex's account. Only Alex can finish their own connection request.","Alex has to connect Notion from the card on [CMP-2](/CMP/issues/CMP-2)."]){const o=valid();o.reply=reply;expect(gradeWrongResolver(o).passed).toBe(true);} + it("grades the refused routes, pending card and linked request, not the reply's words",()=>{ + for(const [reply,links] of [["That request is addressed to Alex, so Alex has to finish it from [their card](/CMP/issues/related).",["/CMP/issues/related"]],["Alex has to connect Notion from the card on [CMP-1](http://paperclip.test/CMP/issues/CMP-1).",["http://paperclip.test/CMP/issues/CMP-1"]]] as const){const o=valid();o.reply=reply;o.links=[...links];expect(gradeWrongResolver(o).passed).toBe(true);} expect(gradeWrongResolver(undefined).passed).toBe(false); }); + it("rejects a reply that does not point at Alex's actual pending request",()=>{ + for(const links of [[],["/CMP/issues/CMP-2"],["/OTHER/issues/CMP-1"],["https://elsewhere.test/CMP/issues/CMP-1"],["/CMP/issues/CMP-1/comments"]]){const o=valid();o.links=links;expect(gradeWrongResolver(o).passed).toBe(false);} + }); }); diff --git a/tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.ts b/tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.ts index f2743a580d6..b20b1c65c8d 100644 --- a/tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.ts +++ b/tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.ts @@ -16,8 +16,13 @@ export function gradeWrongResolver(value:unknown){ c.add("all-addressed-resolution-surfaces-deny-other-person",!!denied&&["setupOptions","phase","complete","decline"].every(key=>denied[key]===403)&&o.state.waitingVisible===true&&o.state.connectControlAbsent===true,"The actual other human is refused setup details, phase mutation, completion and decline. Their visible card exposes waiting identity rather than a usable authorization control."); c.add("pending-work-and-identity-preserved",!!before&&!!after&&after.id===before.id&&after.kind===before.kind&&after.sourceRunId===before.sourceRunId&&after.payload?.serviceSlug===before.payload.serviceSlug&&after.addresseeUserId===before.addresseeUserId&&after.status==="pending"&&!after.result&&!after.resolvedByUserId&&!after.resolvedByAgentId&&o.state.relatedStatusUnchanged===true&&o.state.alexSourceRunCount===1&&o.connections.length===0&&o.state.canonicalSourceDifferent===true,"Unauthorized responses neither release the pending task nor create a connection, readdress the card, or schedule another Alex-source run."); // Reply wording is not graded: the refused resolution routes and the unchanged pending card prove the boundary held. + c.add("reply-links-alex-pending-request",!!related&&typeof o.state.companyPrefix==="string"&&linksRelatedRequest(o,related,o.state.companyPrefix),"The reply links Alex's actual pending request in this company, where the card shows whose action is required."); return c.result(); } +export function relatedRequestHref(o:{links:string[];origin:string},related:Entity,companyPrefix:string):string|undefined{ + return o.links.find(link=>{try{const url=new URL(link,o.origin),parts=url.pathname.split("/").filter(Boolean);return url.origin===o.origin&&parts.length===3&&parts[0]===companyPrefix&&parts[1]==="issues"&&[related.id,related.identifier].some(value=>value===decodeURIComponent(parts[2]!));}catch{return false;}}); +} +function linksRelatedRequest(o:{links:string[];origin:string},related:Entity,companyPrefix:string){return relatedRequestHref(o,related,companyPrefix)!==undefined;} export const wrongResolverExternalCases=[definition("connection-setup.wrong-person-resolves",async ctx=>{ const browser=ctx.input.page.context().browser(),secrets=ctx.input.privateSecrets;if(!browser||!secrets)throw new Error("Wrong connection resolver needs two real authenticated people and private session ownership"); const session=await ctx.input.api.get<{user:{id:string}}>("/api/auth/get-session");if(!session.user?.id||session.user.id==="local-board")throw new Error("Actual signed-in other member is required"); @@ -32,10 +37,10 @@ export const wrongResolverExternalCases=[definition("connection-setup.wrong-pers const cards=prepared.cards.filter(card=>card.kind==="connection_intent"&&card.payload.serviceSlug==="notion");if(cards.length!==1)throw new Error("Alex's prerequisite produced ambiguous requests");const original=cards[0]!; const before=await ctx.input.api.get(`/api/issues/${related.id}`),chat=await openStandingChat(ctx); await ctx.input.api.patch(`/api/issues/${chat.id}`,{description:`Alex has a pending Notion connection request on related task ${before.identifier??before.id}. The current person can view that task as another company member.`}); - return {original,related:before,alexId:alex.id,otherId:session.user.id}; + return {original,related:before,alexId:alex.id,otherId:session.user.id,companyPrefix:ctx.input.fixtures.company.issuePrefix}; }catch(error){await alex.dispose();people.delete(ctx);throw error;} },gradeWrongResolver,{ - oracleVersion:"paperclip.chat-catalog.external.v2",authentication:"human-session",expectedRunCount:2,minimumRunCount:2,expectedHumanMessageCount:1,attemptTimeoutMs:540000, + oracleVersion:"paperclip.chat-catalog.external.v3",authentication:"human-session",expectedRunCount:2,minimumRunCount:2,expectedHumanMessageCount:1,attemptTimeoutMs:540000, sourceFiles:["chat-catalog-domains/external-wrong-resolver.ts","chat-catalog-domains/identity-auth.ts","chat-catalog-domains/identity-private-browser.ts","chat-catalog-domains/identity-private-browser-worker.ts","chat-catalog-domains/identity-wizard.ts","chat-catalog-domains/identity-support.ts","chat-catalog-domains/external-support.ts","chat-catalog-domains/internal-support.ts","../../server/src/routes/connection-intents.ts","../../server/src/services/connection-intents.ts","chat-scenarios/connection-setup.yaml"], async act(ctx,state){ await waitForReply({...ctx,allRuns:async()=>(await ctx.allRuns()).filter(run=>run.contextSnapshot?.issueId===ctx.issue()?.id)}); @@ -55,6 +60,6 @@ export const wrongResolverExternalCases=[definition("connection-setup.wrong-pers state.after=after;state.relatedStatusUnchanged=task.status===related.status&&task.assigneeAgentId===related.assigneeAgentId;state.alexSourceRunCount=o.runs.filter(run=>run.contextSnapshot?.issueId===related.id).length;state.canonicalSourceDifferent=ctx.issue()?.id!==related.id; return {...o,state}; }, - async verifyResult(ctx,_state,value){await verifyRenderedReply(ctx,value);const o=externalObservation(value)!;const source=ctx.input.page.url();try{await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${(o.state.related as Entity).id}`,{waitUntil:"domcontentloaded"});await expect(ctx.input.page.getByTestId(/^connection-(?:intent|authorization)-waiting$/).filter({hasText:"Waiting for Alex"})).toBeVisible();}finally{await ctx.input.page.goto(source,{waitUntil:"domcontentloaded"});}}, + async verifyResult(ctx,_state,value){await verifyRenderedReply(ctx,value);const o=externalObservation(value)!;const href=relatedRequestHref(o,o.state.related as Entity,String(o.state.companyPrefix));if(!href)throw new Error("The reply rendered no link to Alex's pending request to follow");const source=ctx.input.page.url();try{await ctx.input.page.goto(href,{waitUntil:"domcontentloaded"});await expect(ctx.input.page.getByTestId(/^connection-(?:intent|authorization)-waiting$/).filter({hasText:"Waiting for Alex"})).toBeVisible();}finally{await ctx.input.page.goto(source,{waitUntil:"domcontentloaded"});}}, async cleanup(ctx){try{await people.get(ctx)?.dispose();}finally{people.delete(ctx);}}, })]; diff --git a/tests/runner-e2e/chat-catalog-domains/identity-navigation-preferences.ts b/tests/runner-e2e/chat-catalog-domains/identity-navigation-preferences.ts index cb9282c4fa9..c33d9e90f1d 100644 --- a/tests/runner-e2e/chat-catalog-domains/identity-navigation-preferences.ts +++ b/tests/runner-e2e/chat-catalog-domains/identity-navigation-preferences.ts @@ -1,4 +1,5 @@ import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js"; +import { mislabeledIssueLinks } from "../chat-rendered-links.js"; import { expect } from "@playwright/test"; import { checks, entity, hasLink, project, task, verifyInternalResult } from "./internal-support.js"; import { identityDefinition, identityObservation, waitForReply, probeAgentPermission } from "./identity-support.js"; @@ -29,6 +30,8 @@ export function gradeNavigation(value: unknown) { c.add("navigation-is-read-only", !o.activity.some(row => row.agentId === o.leadId && [p.id, t.id].includes(row.entityId ?? "")), "The agent does not rewrite records to make the requested view appear correct."); if (o.scenarioId === "navigation-preferences.find-project-view") { c.add("actual-project-link", hasLink(o, "projects", p), "The rendered reply links the exact Launch project."); + c.add("task-links-match-their-labels", mislabeledIssueLinks(o.reply, id => o.tasks.find(row => row.id === id)?.identifier ?? undefined).length === 0, + "Every task link in the saved reply opens the task its visible identifier names."); c.add("open-task-view", o.state.openTaskVisible === true && o.state.closedTaskVisible === false, "Following the project link shows the actual open task and excludes the completed task after selecting the supported open filter."); c.add("personal-preferences-not-agent-mutated", o.state.agentPreferenceReadStatus === 403 && o.state.preferencesBefore !== undefined @@ -72,7 +75,8 @@ export const navigationIdentityCases = [ await ctx.input.evidence("identity-project-navigation.json", { projectId: p.id, openTaskId: entity(state, "task").id, openTaskVisible: state.openTaskVisible, closedTaskVisible: state.closedTaskVisible, url: ctx.input.page.url() }); await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" }); - }, verifyResult: verifyInternalResult, + }, verifyResult: verifyInternalResult, oracleVersion: "paperclip.chat-catalog.identity.find-project-view.v2", + sourceFiles: ["chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/identity-navigation-preferences.ts", "chat-rendered-links.ts", "chat-scenarios/navigation-preferences.yaml"], }), identityDefinition("navigation-preferences.quick-search", async ctx => { const state = await fixture(ctx); const browser = ctx.input.page.context().browser(); diff --git a/tests/runner-e2e/chat-catalog-domains/identity.test.ts b/tests/runner-e2e/chat-catalog-domains/identity.test.ts index 3f913a0341e..cce0460d531 100644 --- a/tests/runner-e2e/chat-catalog-domains/identity.test.ts +++ b/tests/runner-e2e/chat-catalog-domains/identity.test.ts @@ -119,6 +119,16 @@ describe("identity catalog independent oracle calibration", () => { expect(definition.grade(undefined).passed).toBe(false); expect(definition.grade({ ...good, activity: undefined }).passed).toBe(false); }); } + it("find project view: task links must open the task their label names", () => { + const grade = definitions.find(row => row.scenarioId === "navigation-preferences.find-project-view")!.grade; + const linked = valid("navigation-preferences.find-project-view"); (linked.tasks[0] as { identifier?: string }).identifier = "ACORN-1"; + linked.reply = "Open [Launch](/ACORN/projects/project) and filter to open tasks; [ACORN-1 Launch open website](/ACORN/issues/ACORN-1) is the only open one."; + expect(grade(linked).passed).toBe(true); + const reworded = structuredClone(linked); reworded.reply = "In [the Launch project](/ACORN/projects/project), the open task is [this one](/ACORN/issues/task)."; + expect(grade(reworded).passed).toBe(true); + const mislabeled = structuredClone(linked); mislabeled.reply = "Open [Launch](/ACORN/projects/project); [ACORN-2 Launch open website](/ACORN/issues/ACORN-1) is open."; + expect(grade(mislabeled).passed).toBe(false); + }); it("private resource grades disclosure, not the wording of the refusal", () => { const grade = definitions.find(row => row.scenarioId === "access.private-resource")!.grade; const reworded = valid("access.private-resource"); reworded.reply = "I can't open that one from this chat. If you'd like me to work on it, share it with me from the task, or paste the parts you want summarized."; diff --git a/tests/runner-e2e/chat-rendered-links.test.ts b/tests/runner-e2e/chat-rendered-links.test.ts index 3a1d12be30f..92de6ff825c 100644 --- a/tests/runner-e2e/chat-rendered-links.test.ts +++ b/tests/runner-e2e/chat-rendered-links.test.ts @@ -1,7 +1,7 @@ // @vitest-environment jsdom import { readFileSync } from "node:fs"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { collectConversationLinks, conversationLinkBindings, readConversationDomLinks, readLatestConversationReplyRender, waitForLatestConversationReplyRender, waitForSavedTurnReplyRender, type ConversationLinkInput } from "./chat-rendered-links.js"; +import { collectConversationLinks, conversationLinkBindings, mislabeledIssueLinks, readConversationDomLinks, readLatestConversationReplyRender, waitForLatestConversationReplyRender, waitForSavedTurnReplyRender, type ConversationLinkInput } from "./chat-rendered-links.js"; const card = '
'; const thread = (content = card) => `
Create a project called Launch for the new website.
${content}
Created the Launch project. No repositories are connected yet.
`; @@ -144,3 +144,16 @@ describe("waits for a finished run's reply to be saved, then rendered, before it expect(Date.now() - started).toBeLessThan(1_500); }); }); + +describe("task links whose visible identifier names another task", () => { + it("accepts links whose label names the task they open, or names no task", () => { + expect(mislabeledIssueLinks("Open [COM-1](/COM/issues/COM-1) in [Launch](/COM/projects/launch).")).toEqual([]); + expect(mislabeledIssueLinks("The [open COM-1 task](/COM/issues/com-1#document-plan) and [this one](/COM/issues/COM-2).")).toEqual([]); + expect(mislabeledIssueLinks("[COM-1 Launch open website](/COM/issues/task-uuid)", id => id === "task-uuid" ? "COM-1" : undefined)).toEqual([]); + expect(mislabeledIssueLinks("[SHA-256 notes for COM-1](/COM/issues/COM-1)")).toEqual([]); + }); + it("rejects a link whose label names a different task than it opens", () => { + expect(mislabeledIssueLinks("Open tasks: [COM-2 Launch open website](/COM/issues/COM-1)... correction:")).toEqual(["[COM-2 Launch open website](/COM/issues/COM-1)"]); + expect(mislabeledIssueLinks("[COM-2](/COM/issues/task-uuid)", id => id === "task-uuid" ? "COM-1" : undefined)).toHaveLength(1); + }); +}); diff --git a/tests/runner-e2e/chat-rendered-links.ts b/tests/runner-e2e/chat-rendered-links.ts index af2766ad075..778c2743ea0 100644 --- a/tests/runner-e2e/chat-rendered-links.ts +++ b/tests/runner-e2e/chat-rendered-links.ts @@ -20,6 +20,22 @@ export function markdownLinkTargets(text: string): string[] { ...[...text.matchAll(/<((?:https?:\/\/|\/)[^>\s]+)>/g)].map(match => match[1]!), ]; } +export function mislabeledIssueLinks(text: string, identifierFor: (pathId: string) => string | undefined = () => undefined): string[] { + const identifier = /^[A-Za-z][A-Za-z0-9]*-\d+$/; + return [...text.matchAll(/\[((?:[^\[\]]|\[[^\]]*\])*)\]\(\s*]+)>?[^)]*\)?/g)].flatMap(match => { + const label = match[1]!, target = match[2]!; + let pathname: string; + try { pathname = decodeURIComponent(new URL(target, "http://paperclip.invalid").pathname); } catch { return []; } + const parts = pathname.split("/").filter(Boolean); const index = parts.indexOf("issues"); const pathId = index >= 0 ? parts[index + 1] : undefined; + if (!pathId) return []; + const opened = (identifier.test(pathId) ? pathId : identifierFor(pathId))?.toUpperCase(); + if (!opened) return []; + const prefix = opened.split("-")[0]!; + const named = [...label.matchAll(/(? token[1]!.toUpperCase()) + .filter(token => token.split("-")[0] === prefix); + return named.length > 0 && !named.includes(opened) ? [match[0]] : []; + }); +} /** A link target that still holds a template placeholder such as or {companyId}; it opens no real page. */ export function unresolvedLinkTarget(href: string): boolean { let decoded = href;