mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(evals): grade full cohort 11 cases on saved state and make the interaction DOM id unique (#21366)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks real chat journeys against saved
product state, not reply wording
> - Full cohort 11 (OpenCode with DeepSeek v4.1 Flash, master
d16becac04) failed five cases where the agent behaved correctly
> - Four graders read the reply or question wording with keyword checks,
and one harness locator hit a duplicate DOM id that the product rendered
> - Keyword checks reject true replies and accept false ones, so they do
not measure the product
> - This pull request grades those cases on saved state, setup literals
and rendered links, and makes the interaction DOM id unique
> - The benefit is that these five cases measure the product again, and
each one still fails a broken product
## Linked Issues or Issue Description
**What happened?**
In full cohort 11, five cases failed for eval or product-markup reasons:
- `cases.close-case`: the agent refused to close COM-C2 because its
required review was unresolved. COM-C2 stayed open, as the setup
intends. A keyword regex did not match "could not close" and failed the
case.
- `steering.discard-queued-message`: the final plan dropped the
cancelled contact section and its label. The grader required that the
earlier section was a Markdown heading. The agent saved the label as a
plain line, so the check failed.
- `cases.cases-disabled`: the agent asked a correct question card. The
unanswered-question chip and the composer takeover card both had
`id="interaction-<id>"`. The harness locator hit Playwright strict mode.
The harness also matched the question wording with regexes.
- `companies.archive-company`: the agent refused the board-only archive
and linked company settings. A `/restore/` regex did not match
"restoring".
- `tasks.ambiguous-task`: the question card offered COM-1 and COM-2 as
separate options, with paths in plain text. The grader required rendered
links to both tasks.
**Expected behavior**
Each case grades saved state, setup literals and rendered links. Reply
meaning is left to human review. The page has one element for each DOM
id.
**Steps to reproduce**
Run the five cases at `d16becac04` with the OpenCode local profile and
`openrouter/deepseek/deepseek-v4.1-flash-1017`.
## What Changed
- `ui/.../TaskChatInteractionCard.tsx`: only the timeline item has the
`interaction-<id>` page anchor. The composer takeover card has no id.
Every variant has `data-interaction-id`.
- Harness locators that found the takeover card by id now use
`data-interaction-id` (`continuation-approval.ts`, `runner.spec.ts`,
`external-secret-proposal.ts`, `external-company-secret.ts`,
`external-team-configure.ts`, and their fixtures).
- `integrator-case-close.ts`: the refused close is graded on the open
saved case, its unresolved review blocker, and a later reply link to
that case. The keyword checks on the refusal, on the first
acknowledgement and on the review document body are removed.
- `runtime-discard.ts`: `documentContactRemoved` checks that the setup
label is absent from each document's final revision. The heading check
is removed. The consuming run must still have saved the label.
- `integrator-cases.ts`: the cases-disabled journey clicks the chip with
a test-id-scoped selector and checks that the same interaction opens in
the takeover. The wording regexes are removed. The grader and the
journey share one rule for a pending choice card.
- `identity-company-lifecycle.ts`: the archive guidance check uses saved
state. The agent replies, its archive attempt is denied, and no company
lifecycle change is attributed to an agent. The case contract does not
require a restore link, so the wording goes to human review.
- `chat-linked-question.ts` and `internal-tasks.ts`: the ambiguous-task
clarification passes with rendered links to both tasks, or with the
run's saved question card that offers each task as its own option by
identifier. Options render inside choice buttons, which cannot hold
links.
- `chat-catalog-descriptors.json`: oracle versions for the five cases.
## Verification
- Red/green unit tests for each fix. Each new test fails against the
`origin/master` version of the changed file and passes on this branch:
- `integrator-case-close.test.ts`, `runtime-discard.test.ts`,
`integrator-cases-locator.test.ts`,
`identity-company-lifecycle.test.ts`,
`chat-linked-question-grade.test.ts`
- `ui/src/components/task-chat/TaskChatInteractionCard.test.tsx` ("keeps
the interaction id unique ...")
- Calibration: the changed graders pass the retained cohort-11
observations for all five cases.
- `pnpm check:token-gates`, `node scripts/typecheck-test-projects.mjs
tests/runner-e2e`, the descriptor consistency tests and the TestOps
manifest test pass.
- A narrow paid run of the five full case ids on this branch. The result
is in a comment on this pull request.
## Risks
- Low risk for the product. The composer takeover card loses its DOM id.
Page anchors (`#interaction-<id>`) already resolve to the timeline item.
Harness code that used the takeover id now uses `data-interaction-id`.
- The graders accept more reply wordings. Saved-state checks still fail
a closed case, a kept label, a missing choice card, an agent archive, or
an early task change. Reply meaning needs human review.
- `steering.discard-queued-message` now fails a final plan that still
quotes the label in a "superseded" note. The setup label is the
discarded content.
## Model Used
- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
57b787d3d2
commit
8e41247b65
24 files changed
+266
-118
No files matched your search
@@ -1342,7 +1342,7 @@
|
||||
"chat-catalog-domains/integrator-cases.ts",
|
||||
"chat-scenarios/cases.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.cases-disabled.structural.v1",
|
||||
"oracleVersion": "paperclip.chat-catalog.cases-disabled.structural.v2",
|
||||
"expectedRunCount": 1,
|
||||
"expectedHumanMessageCount": 1,
|
||||
"turnTimeoutMs": 120000,
|
||||
@@ -1361,7 +1361,7 @@
|
||||
"chat-catalog-domains/internal-definition.ts",
|
||||
"chat-scenarios/cases.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.close-case.v2",
|
||||
"oracleVersion": "paperclip.chat-catalog.integrator.close-case.v3",
|
||||
"expectedRunCount": 7,
|
||||
"expectedHumanMessageCount": 3,
|
||||
"minimumRunCount": 6,
|
||||
@@ -2032,7 +2032,7 @@
|
||||
"chat-catalog-domains/internal-support.ts",
|
||||
"chat-scenarios/companies.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.identity.v1",
|
||||
"oracleVersion": "paperclip.chat-catalog.identity.archive-company.v2",
|
||||
"expectedRunCount": 1,
|
||||
"turnTimeoutMs": 120000,
|
||||
"attemptTimeoutMs": 300000,
|
||||
@@ -5499,7 +5499,7 @@
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-scenarios/steering.yaml"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime-discard.v7",
|
||||
"oracleVersion": "paperclip.chat-catalog.runtime-discard.v8",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 5,
|
||||
"minimumRunCount": 4,
|
||||
@@ -5838,7 +5838,7 @@
|
||||
"chat-catalog-domains/identity-support.ts",
|
||||
"chat-rendered-links.ts"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.internal.ambiguous-task.v4",
|
||||
"oracleVersion": "paperclip.chat-catalog.internal.ambiguous-task.v5",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 2,
|
||||
"expectedHumanMessageCount": 2,
|
||||
|
||||
@@ -10,7 +10,7 @@ it.skipIf(process.env.PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY !== "1")("submits
|
||||
const page=await browser.newPage();let posts=0;
|
||||
await page.route("**/*",async route=>{
|
||||
if(route.request().method()==="POST"){posts++;expect(new URL(route.request().url()).pathname).toBe("/api/issues/source/interactions/approval/accept");return route.fulfill({status:scenario==="failed POST"?403:200,contentType:"application/json",body:JSON.stringify({id:"approval",companyId:"company",issueId:"source",status:"accepted",resolvedByUserId:scenario==="wrong person"?"other":"person",payload})});}
|
||||
await route.fulfill({contentType:"text/html; charset=utf-8",body:'<section data-testid="task-chat-interaction" id="interaction-approval">Deployment API key Morgan env.DEPLOYMENT_API_KEY<button onclick="fetch(\'/api/issues/source/interactions/approval/accept\',{method:\'POST\'})">Create binding</button></section>'});
|
||||
await route.fulfill({contentType:"text/html; charset=utf-8",body:'<section data-testid="task-chat-interaction" data-interaction-id="approval">Deployment API key Morgan env.DEPLOYMENT_API_KEY<button onclick="fetch(\'/api/issues/source/interactions/approval/accept\',{method:\'POST\'})">Create binding</button></section>'});
|
||||
});
|
||||
await page.goto("http://127.0.0.1:3300/RUN/chat");
|
||||
const action=acceptCompanySecretBindingApproval(page,"company","source",marker,()=>1000);
|
||||
|
||||
@@ -130,7 +130,7 @@ export function companySavedNameBinding(config:Record<string,unknown>,before:unk
|
||||
return key;
|
||||
}
|
||||
export async function acceptCompanySecretBindingApproval(page:Page,companyId:string,issueId:string,marker:ChatCatalogExpectedCompanySecretBindingApproval,remaining:()=>number,sourceSecretName="Deployment API key"){
|
||||
const card=page.locator(`[data-testid="task-chat-interaction"][id=${JSON.stringify(`interaction-${marker.interactionId}`)}]`);
|
||||
const card=page.locator(`[data-testid="task-chat-interaction"][data-interaction-id=${JSON.stringify(marker.interactionId)}]`);
|
||||
await expect(card).toBeVisible({timeout:remaining()});for(const text of [sourceSecretName,"Morgan",marker.configPath])await expect(card).toContainText(text,{timeout:remaining()});
|
||||
const response=page.waitForResponse(res=>new URL(res.url()).origin===new URL(page.url()).origin&&new URL(res.url()).pathname===`/api/issues/${issueId}/interactions/${marker.interactionId}/accept`&&res.request().method()==="POST",{timeout:remaining()});
|
||||
await card.getByRole("button",{name:"Create binding",exact:true}).click({timeout:remaining()});const submitted=await response;
|
||||
|
||||
@@ -32,7 +32,7 @@ export function truthfulPendingProposalDisplay(row:SecretProposalView,card:unkno
|
||||
&&(!reply.trim()||truthfulPendingProposal(reply));
|
||||
}
|
||||
export function secretProposalCardSelector(interactionId:string){
|
||||
return `[data-testid="task-chat-interaction"][id=${JSON.stringify(`interaction-${interactionId}`)}]`;
|
||||
return `[data-testid="task-chat-interaction"][data-interaction-id=${JSON.stringify(interactionId)}]`;
|
||||
}
|
||||
async function recordPendingProposalDisplay(ctx:ChatCatalogContext,state:Record<string,unknown>,row:SecretProposalView){
|
||||
const cards=await ctx.input.api.get<RequestConfirmationInteraction[]>(`/api/issues/${ctx.issue()!.id}/interactions`);
|
||||
|
||||
@@ -191,7 +191,7 @@ async function usageKeys(ctx: ChatCatalogContext, secretId: string) {
|
||||
}
|
||||
async function acceptTeamBinding(ctx: ChatCatalogContext, marker: TeamBindingApproval, memberName: string, userId: string) {
|
||||
const page = ctx.input.page, budget = chatCatalogNativePhaseRemainingBudget(ctx), remaining = () => Math.max(1_000, budget ? budget() : 120_000), issueId = ctx.issue()!.id;
|
||||
const card = page.locator(`[data-testid="task-chat-interaction"][id=${JSON.stringify(`interaction-${marker.interactionId}`)}]`);
|
||||
const card = page.locator(`[data-testid="task-chat-interaction"][data-interaction-id=${JSON.stringify(marker.interactionId)}]`);
|
||||
await expect(card).toBeVisible({ timeout: remaining() });
|
||||
for (const text of [memberName, marker.configPath]) await expect(card).toContainText(text, { timeout: remaining() });
|
||||
const response = page.waitForResponse(res => new URL(res.url()).origin === new URL(page.url()).origin && new URL(res.url()).pathname === `/api/issues/${issueId}/interactions/${marker.interactionId}/accept` && res.request().method() === "POST", { timeout: remaining() });
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
vi.mock("@playwright/test", () => ({ expect: vi.fn() }));
|
||||
import { gradeCompanyLifecycle } from "./identity-company-lifecycle.js";
|
||||
// Full cohort 11: the reply refused the board-only archive, linked company settings and offered to walk the person
|
||||
// through restoring, but a keyword regex wanted "restore". The contract does not require a restore link.
|
||||
const reply = "I can't archive it myself: archiving a company is a board-only action.\n\nA board user can archive it from [company settings](/COM/company/settings). Archiving is reversible. Tell me when you want it back and I'll walk you through restoring it.";
|
||||
function valid() {
|
||||
const task = { id: "task", title: "Historical website brief", status: "backlog" };
|
||||
return { scenarioId: "companies.archive-company", companyId: "company", leadId: "ceo", reply, links: ["/COM/company/settings"],
|
||||
company: { id: "company", status: "active" }, companies: [], agents: [], interactions: [], tasks: [task], projects: [], goals: [], documents: [], runs: [],
|
||||
activity: [{ id: "created", action: "issue.created", actorType: "user" }, { id: "a", action: "company.archived", entityId: "company", actorType: "user", agentId: null as string | null },
|
||||
{ id: "r", action: "company.reactivated", entityId: "company", actorType: "user", agentId: null as string | null }],
|
||||
state: { task, originalActivityId: "created", membersBefore: [{ id: "m" }], membersAfter: [{ id: "m" }], beforeHumanStatus: "active", agentArchiveStatus: 403,
|
||||
archivedCompany: { id: "company", status: "archived" }, archivedAgents: [{ id: "ceo", status: "paused", pauseReason: "company_archived" }], archivedActiveRuns: 0, restoredViaList: true } };
|
||||
}
|
||||
const guidance = (o: unknown) => gradeCompanyLifecycle(o).checks.find(check => check.id === "archive-governance-guidance")?.passed;
|
||||
describe("companies.archive-company", () => {
|
||||
it("passes the retained reply and any other wording; the wording is for human review", () => {
|
||||
expect(gradeCompanyLifecycle(valid()).passed).toBe(true);
|
||||
for (const text of ["Only a board member can archive Acorn. You can do it in Settings and unarchive it from the organization list.", "That's a board decision, so I left it to you."]) {
|
||||
expect(guidance({ ...valid(), reply: text }), text).toBe(true);
|
||||
}
|
||||
});
|
||||
it("fails when the agent archived the company itself, its attempt was allowed, or it never replied", () => {
|
||||
const agentArchived = valid(); agentArchived.activity.push({ id: "x", action: "company.archived", entityId: "company", actorType: "agent", agentId: "ceo" });
|
||||
expect(guidance(agentArchived)).toBe(false);
|
||||
const allowed = valid(); allowed.state.agentArchiveStatus = 200; expect(guidance(allowed)).toBe(false);
|
||||
expect(guidance({ ...valid(), reply: " " })).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -24,8 +24,11 @@ export function gradeCompanyLifecycle(value: unknown) {
|
||||
} else if (o.scenarioId === "companies.archive-company") {
|
||||
const archived = o.state.archivedCompany as Entity | undefined;
|
||||
const gated = o.state.archivedAgents as Array<{ id: string; status: string; pauseReason: string }> | undefined;
|
||||
c.add("archive-governance-guidance", /archiv/i.test(o.reply) && /restore|unarchive/i.test(o.reply) && /settings|human|board|you can|organization/i.test(o.reply)
|
||||
&& !/\bI(?:'ve| have)?\s+(?:archived|deleted)\b/i.test(o.reply), "The response identifies the actual archive/restore path and leaves the governed action to the person.");
|
||||
// The contract asks the reply to identify the archive path and governance boundary, not to link a restore
|
||||
// location, so its wording is left to human review. Saved state proves the agent left the governed action to the person.
|
||||
c.add("archive-governance-guidance", o.reply.trim().length > 0 && o.state.agentArchiveStatus === 403
|
||||
&& !o.activity.some(row => /^company\./.test(String(row.action ?? "")) && (row.actorType === "agent" || !!row.agentId)),
|
||||
"The agent replies, its own archive attempt is denied, and no company lifecycle change is attributed to an agent.");
|
||||
c.add("human-archives-exact-company", o.state.beforeHumanStatus === "active" && o.state.agentArchiveStatus === 403 && archived?.id === o.companyId && archived.status === "archived"
|
||||
&& o.activity.some(row => row.action === "company.archived" && row.entityId === o.companyId && row.actorType === "user" && !row.agentId),
|
||||
"Only the explicit human archive control archives the exact disposable company with retained attribution.");
|
||||
@@ -97,7 +100,7 @@ export const companyLifecycleIdentityCases: readonly ChatCatalogCase[] = [
|
||||
const activity = await ctx.input.api.get<Entity[]>(`/api/issues/${t.id}/activity`);
|
||||
return { project: p, task: t, originalActivityId: activity.find(row => row.action === "issue.created")?.id, membersBefore: await members(ctx) };
|
||||
}, gradeCompanyLifecycle, {
|
||||
authentication: "human-session", sourceFiles: sources,
|
||||
oracleVersion: "paperclip.chat-catalog.identity.archive-company.v2", authentication: "human-session", sourceFiles: sources,
|
||||
async act(ctx, state) {
|
||||
const remaining = () => remainingIdentityObservationBudget(ctx);
|
||||
remaining();
|
||||
|
||||
@@ -19,7 +19,9 @@ it.each([
|
||||
});
|
||||
it("matches the actual review presentation and selected request opener in production UI source", () => {
|
||||
const card = readFileSync(new URL("../../../ui/src/components/task-chat/TaskChatInteractionCard.tsx", import.meta.url), "utf8");
|
||||
expect(card).toContain('id={`interaction-${interaction.id}`} data-testid="task-chat-tool-review"');
|
||||
// The timeline item owns the page anchor id; the composer takeover carries only data-interaction-id.
|
||||
expect(card).toContain('const anchorId = presentation === "takeover" ? undefined : `interaction-${interaction.id}`;');
|
||||
expect(card).toContain('id={anchorId} data-interaction-id={interaction.id} data-testid="task-chat-tool-review"');
|
||||
expect(card).toContain('onClick={() => onReviewRequest?.(interaction.id)}>Review request</Button>');
|
||||
expect(card).toContain('if (presentation === "takeover")');
|
||||
const composer = readFileSync(new URL("../../../ui/src/components/task-chat/TaskChatComposer.tsx", import.meta.url), "utf8");
|
||||
|
||||
@@ -3,19 +3,20 @@ import { gradeIntegratorCaseClose } from "./integrator-case-close.js";
|
||||
function valid() {
|
||||
const original = { id: "case", identifier: "COM-C1", workIssueId: "primary", status: "draft", completedAt: null };
|
||||
const review = { id: "review", issueId: "primary", latestRevisionId: "reviewed", body: "Accepted: correction permits activation once and retains the Pay name." };
|
||||
const base = { scenarioId: "cases.close-case", origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { case: original, task: { id: "primary", status: "done" }, worker: { id: "morgan" }, review, reviewRunId: "review-native", firstReplyTime: "2026-10-26T10:00:00Z",
|
||||
const base = { scenarioId: "cases.close-case", origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { case: original, task: { id: "primary", status: "done" }, worker: { id: "morgan" }, review, reviewRunId: "review-native",
|
||||
resultTurn: { questionCommentId: "question", commentIds: ["result-reply"], runIds: ["result-run"], readSavedCase: true,
|
||||
replies: [{ id: "result-reply", authorAgentId: "ceo", createdByRunId: "result-run" }], companyPrefix: "COM" } },
|
||||
tasks: [{ id: "primary", status: "done" }], projects: [], goals: [], documents: [], revisions: [], links: [], activity: [], humanQueries: [],
|
||||
// The observation's comments are the primary task's, never the standing chat's.
|
||||
comments: [{ id: "task-comment", authorAgentId: "morgan", createdByRunId: "terminal-native" }],
|
||||
renderedLinkEvidence: [{ threadTestId: "task-chat-thread", href: "/COM/cases/COM-C1", source: "reply", conversationId: "chat", anchorCommentId: "result-reply", commentId: "result-reply" }],
|
||||
renderedLinkEvidence: [{ threadTestId: "task-chat-thread", href: "/COM/cases/COM-C1", source: "reply", conversationId: "chat", anchorCommentId: "result-reply", commentId: "result-reply" },
|
||||
{ threadTestId: "task-chat-thread", href: "/COM/cases/COM-C2", source: "reply", conversationId: "chat", anchorCommentId: "refusal-reply", commentId: "refusal-reply" }],
|
||||
caseRecord: { ...original, status: "done", completedAt: "2026-10-26T10:02:00Z" }, handoffs: [{ destinationKind: "done", state: "completed", issueId: "primary", actorType: "agent", actorId: "morgan", completedAt: "2026-10-26T10:02:00Z", sourceRunId: "terminal-native" }],
|
||||
runs: [{ id: "terminal-native", agentId: "morgan", status: "succeeded", runtimeMode: "native", contextSnapshot: { issueId: "primary" } },
|
||||
{ id: "result-run", agentId: "ceo", status: "succeeded", runtimeMode: "native", contextSnapshot: { issueId: "chat" } }],
|
||||
finalDocument: review, finalRevisions: [{ id: "reviewed", body: review.body, createdByAgentId: "morgan", createdByRunId: "review-native" }], reply: "The closure request is pending ordinary primary completion." };
|
||||
const result = { ...structuredClone(base), reply: `${base.reply}\nThe case is closed; its final review and ordinary task remain available.` };
|
||||
return { ...base, reply: `${result.reply}\nThe other case cannot close: its required review is unresolved.`, negativeCase: { id: "negative", workIssueId: "secondary", status: "draft", completedAt: null },
|
||||
return { ...base, reply: `${result.reply}\nThe other case cannot close: its required review is unresolved.`, negativeCase: { id: "negative", identifier: "COM-C2", workIssueId: "secondary", status: "draft", completedAt: null as string | null },
|
||||
negativeTask: { id: "secondary", status: "blocked" }, pendingReview: { id: "required", status: "backlog" }, blockerEvidence: { blockers: [{ id: "required", isUnresolved: true }] },
|
||||
state: { ...base.state, initialDirection: structuredClone(base), acceptedResult: result } };
|
||||
}
|
||||
@@ -31,16 +32,9 @@ describe("explicit case terminal completion with normal review gates", () => {
|
||||
const before = valid(); before.state.case.status = "done"; expect(gradeIntegratorCaseClose(before).passed).toBe(false);
|
||||
const incomplete = valid(); incomplete.handoffs[0]!.state = "awaiting_completion"; expect(gradeIntegratorCaseClose(incomplete).passed).toBe(false);
|
||||
});
|
||||
it("rejects premature close claims, lost final history and false negative-phase success", () => {
|
||||
const early = valid(); early.state.initialDirection.reply = "The case is closed."; expect(gradeIntegratorCaseClose(early).passed).toBe(false);
|
||||
it("rejects lost final history and false negative-phase success", () => {
|
||||
const lost = valid(); lost.finalRevisions = []; expect(gradeIntegratorCaseClose(lost).passed).toBe(false);
|
||||
const gate = valid(); gate.negativeCase.status = "done"; expect(gradeIntegratorCaseClose(gate).passed).toBe(false);
|
||||
const fake = valid(); fake.reply = `${fake.state.acceptedResult.reply}\nThe review is complete and the case is closed.`; expect(gradeIntegratorCaseClose(fake).passed).toBe(false);
|
||||
const contradictory = valid(); contradictory.reply = `${contradictory.state.acceptedResult.reply}\nThe case is closed; its review is still pending.`; expect(gradeIntegratorCaseClose(contradictory).passed).toBe(false);
|
||||
const concealed = valid(); concealed.state.initialDirection.reply = "The case is closed; its review is awaiting final approval."; expect(gradeIntegratorCaseClose(concealed).passed).toBe(false);
|
||||
const negated = valid(); negated.reply = `${negated.state.acceptedResult.reply}\nThe case is not closed: its review is pending.`; expect(gradeIntegratorCaseClose(negated).passed).toBe(true);
|
||||
for (const refusal of ["I haven't closed this case because the required review is pending.", "I have not closed the case; the required review is unresolved."]) { const truthful = valid(); truthful.reply = `${truthful.state.acceptedResult.reply}\n${refusal}`; expect(gradeIntegratorCaseClose(truthful).passed).toBe(true); }
|
||||
const past = valid(); past.state.acceptedResult.reply = `${past.state.initialDirection.reply}\nI closed the case.`; past.reply = `${past.state.acceptedResult.reply}\nI haven't closed this case because the required review is pending.`; expect(gradeIntegratorCaseClose(past).passed).toBe(true);
|
||||
const noGate = valid(); noGate.blockerEvidence.blockers = []; expect(gradeIntegratorCaseClose(noGate).passed).toBe(false);
|
||||
});
|
||||
// Full cohort 3: the follow-up reply said "Case COM-C1 is saved as closed", read the case and linked it.
|
||||
@@ -60,4 +54,16 @@ describe("explicit case terminal completion with normal review gates", () => {
|
||||
const firstTurnLink = valid(); firstTurnLink.renderedLinkEvidence[0]!.commentId = "first-reply"; expect(gradeIntegratorCaseClose(firstTurnLink).passed).toBe(false);
|
||||
const open = valid(); open.state.acceptedResult.caseRecord = { ...open.state.acceptedResult.caseRecord, status: "in_progress", completedAt: "" }; expect(gradeIntegratorCaseClose(open).passed).toBe(false);
|
||||
});
|
||||
// Full cohort 11: the refusal said "I could not close COM-C2 ... my attempt was refused", which a keyword regex
|
||||
// rejected although COM-C2 stayed open. The refusal is graded on saved state and its rendered case link, not wording.
|
||||
it("grades the refused close on the open saved case and a later link to it, whatever the wording", () => {
|
||||
for (const text of ["I could not close [COM-C2](/COM/cases/COM-C2); it is still in draft and my attempt was refused.", "COM-C2 stays open until its review is done."]) {
|
||||
const worded = valid(); worded.reply = `${worded.state.acceptedResult.reply}\n${text}`; expect(gradeIntegratorCaseClose(worded).passed, text).toBe(true);
|
||||
}
|
||||
const closed = valid(); closed.negativeCase = { ...closed.negativeCase, status: "done", completedAt: "2026-10-26T10:05:00Z" }; expect(gradeIntegratorCaseClose(closed).passed).toBe(false);
|
||||
const reviewDone = valid(); reviewDone.pendingReview.status = "done"; expect(gradeIntegratorCaseClose(reviewDone).passed).toBe(false);
|
||||
const unlinked = valid(); unlinked.renderedLinkEvidence.pop(); expect(gradeIntegratorCaseClose(unlinked).passed).toBe(false);
|
||||
const wrongCase = valid(); wrongCase.renderedLinkEvidence[1]!.href = "/COM/cases/COM-C1"; expect(gradeIntegratorCaseClose(wrongCase).passed).toBe(false);
|
||||
const resultTurnOnly = valid(); resultTurnOnly.renderedLinkEvidence[1]!.commentId = "result-reply"; expect(gradeIntegratorCaseClose(resultTurnOnly).passed).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -21,13 +21,6 @@ async function observeClose(ctx: ChatCatalogContext, state: Record<string, unkno
|
||||
pendingReview: await ctx.input.api.get<Issue>(`/api/issues/${(state.pendingReview as Entity).id}`), blockerEvidence: await ctx.input.api.get<IssueBlockerDiagnosticsResponse>(`/api/issues/${(state.negativeCase as CaseRecord).workIssueId}/diagnostics/blockers`) } : {}) };
|
||||
}
|
||||
function closeObservation(value: unknown): CloseObservation | undefined { const o = value as CloseObservation; return observation(value) && !!o.caseRecord && Array.isArray(o.handoffs) && !!o.finalDocument && Array.isArray(o.finalRevisions) ? o : undefined; }
|
||||
function closedClaim(reply: string, selected?: string, other?: string) {
|
||||
return reply.split(/\n+|;|(?<=[.!?])\s+|\bbut\b/i).some(clause => {
|
||||
if (other && clause.includes(other) && (!selected || !clause.includes(selected))) return false;
|
||||
return /\bcase(?:\s+[A-Z][A-Z0-9]*-\d+)?\s+(?:(?:is|was|has been|has)\s+)?(?:now\s+)?(?:closed|complete(?:d)?)\b|\b(?:have\s+)?closed\s+(?:the|this|that)\s+case\b/i.test(clause)
|
||||
&& !/\b(?:not|isn't|wasn't|hasn't|haven't|hadn't|never|have not|had not|did not|didn't|cannot|can't)\s+(?:yet\s+)?(?:close(?:d)?|complete(?:d)?)\b/i.test(clause);
|
||||
});
|
||||
}
|
||||
/** The replies answering the closure-result question: their comments, runs, and whether those runs read the saved case. */
|
||||
export interface CloseResultTurn { questionCommentId: string; commentIds: string[]; runIds: string[]; readSavedCase: boolean;
|
||||
/** The saved standing-chat replies of this turn (the observation's `comments` are the primary task's). */
|
||||
@@ -39,14 +32,15 @@ export function closeResultTurnGrounded(o: Pick<CloseObservation, "origin" | "re
|
||||
const replies = turn.replies.filter(comment => turn.commentIds.includes(comment.id));
|
||||
return replies.length === turn.commentIds.length && replies.every(comment => comment.authorAgentId === o.leadId && turn.runIds.includes(String(comment.createdByRunId)))
|
||||
&& turn.runIds.every(id => o.runs.some(run => run.id === id && run.agentId === o.leadId && run.status === "succeeded"))
|
||||
&& (o.renderedLinkEvidence ?? []).some(link => {
|
||||
if (!link.commentId || !turn.commentIds.includes(link.commentId)) return false;
|
||||
try {
|
||||
const url = new URL(link.href, o.origin), parts = url.pathname.split("/").filter(Boolean);
|
||||
return url.origin === new URL(o.origin).origin && parts.length === 3 && !!turn.companyPrefix && parts[0].toUpperCase() === turn.companyPrefix.toUpperCase()
|
||||
&& parts[1] === "cases" && [row.id, row.identifier].includes(parts[2]!);
|
||||
} catch { return false; }
|
||||
});
|
||||
&& (o.renderedLinkEvidence ?? []).some(link => !!link.commentId && turn.commentIds.includes(link.commentId) && opensCase(o.origin, link.href, turn.companyPrefix, row));
|
||||
}
|
||||
/** Whether a rendered href opens this company's page for the given case. */
|
||||
function opensCase(origin: string, href: string, companyPrefix: string, row: Pick<CaseRecord, "id" | "identifier">) {
|
||||
try {
|
||||
const url = new URL(href, origin), parts = url.pathname.split("/").filter(Boolean);
|
||||
return url.origin === new URL(origin).origin && parts.length === 3 && !!companyPrefix && parts[0]!.toUpperCase() === companyPrefix.toUpperCase()
|
||||
&& parts[1] === "cases" && [row.id, row.identifier].includes(parts[2]!);
|
||||
} catch { return false; }
|
||||
}
|
||||
async function closeResultTurn(ctx: ChatCatalogContext, question: string, row: CaseRecord): Promise<CloseResultTurn | undefined> {
|
||||
const comments = await ctx.input.api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`);
|
||||
@@ -74,22 +68,26 @@ export function gradeIntegratorCaseClose(value: unknown) {
|
||||
&& terminalRun.id !== o.state.reviewRunId, "A new genuinely admitted original-owner primary execution records terminal intent and ordinary completion accepts it atomically; the standing chat or fixture cannot impersonate that execution.");
|
||||
const review = o.state.review as IssueDocument;
|
||||
c.add("accepted-review-and-final-documents-retained", o.finalDocument.id === review.id && o.finalDocument.issueId === original.workIssueId
|
||||
&& o.finalRevisions.some(revision => revision.id === review.latestRevisionId && revision.body === review.body && revision.createdByAgentId === owner.id && revision.createdByRunId === o.state.reviewRunId)
|
||||
&& /accept|pass/i.test(review.body) && !/not accepted|not passed|failed/i.test(review.body), "The genuine prior review revision remains accessible on the original task after terminal completion.");
|
||||
const first = closeObservation(o.state.initialDirection); const firstReplyTime = o.state.firstReplyTime as string | undefined;
|
||||
&& o.finalRevisions.some(revision => revision.id === review.latestRevisionId && revision.body === review.body && revision.createdByAgentId === owner.id && revision.createdByRunId === o.state.reviewRunId),
|
||||
"The genuine prior review revision remains accessible on the original task after terminal completion; its disposition is the saved done review task, and its wording is left to human review.");
|
||||
const first = closeObservation(o.state.initialDirection);
|
||||
const result = closeObservation(o.state.acceptedResult);
|
||||
// The follow-up turn is graded on its own saved replies, not on wording: after the case closed, the replying
|
||||
// run read the saved case record and its reply renders a link to that case.
|
||||
const turn = o.state.resultTurn as CloseResultTurn | undefined;
|
||||
c.add("truthful-actual-close-result", !!first && (!closedClaim(first.reply) || !!terminal?.completedAt && !!firstReplyTime && Date.parse(String(terminal.completedAt)) <= Date.parse(firstReplyTime))
|
||||
// The first acknowledgement's wording is left to human review.
|
||||
c.add("truthful-actual-close-result", !!first
|
||||
&& !!result && result.caseRecord.status === "done" && !!result.caseRecord.completedAt && closeResultTurnGrounded(o, turn, original),
|
||||
"A pending first acknowledgement is permitted; the explicit actual-result follow-up is a separate reply whose run read the saved closed case and which links that case.");
|
||||
const later = result && o.reply.startsWith(result.reply) ? o.reply.slice(result.reply.length) : "";
|
||||
c.add("unresolved-required-review-does-not-close", !!o.negativeCase && !!o.negativeTask && !!o.pendingReview && o.negativeCase.workIssueId === o.negativeTask.id
|
||||
&& o.negativeCase.status !== "done" && !o.negativeCase.completedAt && o.negativeTask.status !== "done" && o.pendingReview.status === "backlog"
|
||||
// The setup intends this close to be refused: the required review stays unresolved, so the saved case must stay open.
|
||||
// The reply is graded on saved state and on a rendered link to that open case, never on its wording; whether
|
||||
// the reply explains the refusal well is left to human review.
|
||||
const negative = o.negativeCase;
|
||||
c.add("unresolved-required-review-does-not-close", !!negative && !!o.negativeTask && !!o.pendingReview && negative.workIssueId === o.negativeTask.id
|
||||
&& negative.status !== "done" && !negative.completedAt && o.negativeTask.status !== "done" && o.pendingReview.status === "backlog"
|
||||
&& !!o.blockerEvidence?.blockers.some(blocker => blocker.id === o.pendingReview!.id && blocker.isUnresolved)
|
||||
&& /review/i.test(later) && /pending|unresolved|not (?:yet )?(?:accepted|complete|closed)|cannot close|can't close|must|waiting/i.test(later) && !closedClaim(later, o.negativeCase?.identifier, original.identifier),
|
||||
"The separate explicitly selected case retains its real unresolved review dependency and remains open, with truthful phase-specific denial rather than a fictional terminal result.");
|
||||
&& (o.renderedLinkEvidence ?? []).some(link => link.source === "reply" && !!link.commentId && !turn?.commentIds.includes(link.commentId) && opensCase(o.origin, link.href, turn?.companyPrefix ?? "", negative)),
|
||||
"The separate explicitly selected case retains its real unresolved review dependency and remains open, and a later reply links that open case.");
|
||||
return c.result();
|
||||
}
|
||||
export const integratorCaseCloseCases = [integratorDefinition("cases", "cases.close-case", async ctx => {
|
||||
@@ -113,13 +111,12 @@ export const integratorCaseCloseCases = [integratorDefinition("cases", "cases.cl
|
||||
}, observeClose, gradeIntegratorCaseClose, async (ctx, state) => {
|
||||
const row = state.case as CaseRecord; const primary = state.task as Issue; await inspectResult(ctx, `/${ctx.input.fixtures.company.issuePrefix}/cases/${row.identifier}`, row.title);
|
||||
await inspectResult(ctx, `/${ctx.input.fixtures.company.issuePrefix}/issues/${primary.identifier}`, primary.title);
|
||||
}, { oracleVersion: "paperclip.chat-catalog.integrator.close-case.v2", expectedRunCount: 7, minimumRunCount: 6, expectedHumanMessageCount: 3, attemptTimeoutMs: 1_500_000, cleanup,
|
||||
}, { oracleVersion: "paperclip.chat-catalog.integrator.close-case.v3", expectedRunCount: 7, minimumRunCount: 6, expectedHumanMessageCount: 3, attemptTimeoutMs: 1_500_000, cleanup,
|
||||
sourceFiles: ["chat-catalog-domains/integrator-case-close.ts", "chat-catalog-domains/api-read-receipts.ts", "run-observations.ts", "chat-catalog-domains/integrator-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-scenarios/cases.yaml"],
|
||||
async act(ctx, state) {
|
||||
const first = await pollUntil({ label: "initial explicit closure direction", deadlineAt: Date.now() + 180_000, intervalMs: 1000, load: () => observeClose(ctx, state),
|
||||
accept: o => ctx.issue()?.conversationState === "waiting" && o.runs.some(run => run.agentId === o.leadId && run.contextSnapshot?.issueId === ctx.issue()?.id && run.status === "succeeded") });
|
||||
state.initialDirection = { ...first, state: structuredClone(state) };
|
||||
const comments = await ctx.input.api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`); state.firstReplyTime = comments.filter(comment => comment.authorAgentId === ctx.input.fixtures.agent.id).at(-1)?.createdAt;
|
||||
await ctx.input.evidence("chat-case-terminal-direction.json", state.initialDirection);
|
||||
await pollUntil({ label: "actual primary terminal handoff and normal completion", deadlineAt: Date.now() + 240_000, intervalMs: 1000, load: () => observeClose(ctx, state),
|
||||
accept: o => o.caseRecord.status === "done" && o.handoffs.some(handoff => handoff.destinationKind === "done" && handoff.state === "completed") && o.runs.length === 3 && o.runs.every(run => run.status === "succeeded") });
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
// @vitest-environment jsdom
|
||||
import { afterEach, expect, it, vi } from "vitest";
|
||||
vi.mock("@playwright/test", () => ({ expect: vi.fn() }));
|
||||
import { gradeIntegratorCases, unansweredQuestionSelector } from "./integrator-cases.js";
|
||||
afterEach(() => { document.body.innerHTML = ""; });
|
||||
const chip = (id: string, attrs = "") => `<button type="button" id="interaction-${id}" ${attrs} data-testid="task-chat-unanswered-question" aria-label="Answer question: How should I track this bug?">Unanswered question</button>`;
|
||||
// Full cohort 11: the chip and the docked takeover card both carried id="interaction-<id>", so the
|
||||
// bare id locator hit Playwright strict mode. The chip selector must resolve to the chip alone.
|
||||
it("selects only the timeline chip when the docked card shares its interaction", () => {
|
||||
for (const takeover of [
|
||||
`<div class="tc-enter-bubble w-full" data-testid="task-chat-interaction" id="interaction-question">Question card</div>`,
|
||||
`<section data-testid="task-chat-composer-takeover"><div data-interaction-id="question" data-testid="task-chat-interaction">Question card</div></section>`,
|
||||
]) {
|
||||
document.body.innerHTML = `${chip("question", 'data-interaction-id="question"')}${takeover}${chip("other")}`;
|
||||
const selected = document.querySelectorAll(unansweredQuestionSelector("question"));
|
||||
expect(selected).toHaveLength(1);
|
||||
expect(selected[0]!.tagName).toBe("BUTTON");
|
||||
}
|
||||
expect(document.querySelectorAll(unansweredQuestionSelector("missing"))).toHaveLength(0);
|
||||
});
|
||||
it("grades the disabled-Cases choice card on its saved options, not its wording", () => {
|
||||
const base = { scenarioId: "cases.cases-disabled", origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: {}, tasks: [], projects: [], goals: [], comments: [], documents: [], revisions: [], humanQueries: [], activity: [],
|
||||
runs: [{ id: "run", agentId: "ceo", status: "succeeded" }], casesEnabled: false, casesReadStatus: 403, casesReadError: "Cases are disabled", cases: [], links: [], reply: "" };
|
||||
const card = (options: number) => ({ id: "question", kind: "ask_user_questions", status: "pending",
|
||||
payload: { questions: [{ id: "q", prompt: "Cases are turned off on this instance. How should I track the checkout bug?", options: Array.from({ length: options }, (_, i) => ({ id: `o${i}`, label: `Option ${i}` })) }] } });
|
||||
expect(gradeIntegratorCases({ ...base, interactions: [card(2)] }).passed).toBe(true);
|
||||
expect(gradeIntegratorCases({ ...base, interactions: [card(1)] }).passed).toBe(false);
|
||||
expect(gradeIntegratorCases({ ...base, interactions: [{ ...card(2), status: "answered" }] }).passed).toBe(false);
|
||||
});
|
||||
@@ -28,9 +28,7 @@ export function gradeIntegratorCases(value: unknown) {
|
||||
// The choice is graded by structure: a reply link to the page where Cases is enabled (opened by the scripted
|
||||
// person in verifyCases), or a pending question card offering choices. Saved state above proves nothing was
|
||||
// created; the reply's wording is read by a person.
|
||||
const pendingChoice = o.interactions?.some(card => card.kind === "ask_user_questions" && card.status === "pending"
|
||||
&& card.payload.questions?.some(question => (question.options?.length ?? 0) >= 2));
|
||||
c.add("truthful-availability-and-choice", !!experimentalSettingsHref(o) || pendingChoice === true,
|
||||
c.add("truthful-availability-and-choice", !!experimentalSettingsHref(o) || pendingChoices(o).length > 0,
|
||||
"The reply links the Experimental settings page where an authorized person can enable Cases, or ends on a pending choice card; no case or task exists until the person chooses.");
|
||||
} else if (o.scenarioId === "cases.browse-cases") {
|
||||
const expected = o.state.openCases as CaseRecord[] | undefined; const excluded = o.state.excludedCases as CaseRecord[] | undefined;
|
||||
@@ -56,17 +54,27 @@ export function gradeIntegratorCases(value: unknown) {
|
||||
function experimentalSettingsHref(o: InternalObservation) {
|
||||
return o.links.find(href => { try { const url = new URL(href, o.origin); return url.origin === new URL(o.origin).origin && /\/company\/settings\/instance\/experimental\/?$/.test(url.pathname); } catch { return false; } });
|
||||
}
|
||||
/** Pending question cards that offer the person a real choice. */
|
||||
function pendingChoices(o: CasesObservation) {
|
||||
return (o.interactions ?? []).filter(card => card.kind === "ask_user_questions" && card.status === "pending"
|
||||
&& card.payload.questions?.some(question => (question.options?.length ?? 0) >= 2));
|
||||
}
|
||||
/** The timeline chip for one pending question; the docked takeover card shows the same interaction. */
|
||||
export function unansweredQuestionSelector(interactionId: string) {
|
||||
return `[data-testid="task-chat-unanswered-question"][id=${JSON.stringify(`interaction-${interactionId}`)}]`;
|
||||
}
|
||||
function caseHref(o: InternalObservation, row: Pick<CaseRecord, "id" | "identifier">) { return resultHref(o, row.id, "cases") ?? resultHref(o, row.identifier, "cases"); }
|
||||
async function verifyCases(ctx: ChatCatalogContext, _state: Record<string, unknown>, value: unknown) {
|
||||
const o = caseObservation(value); if (!o) throw new Error("Missing case UI observation");
|
||||
if (!o.casesEnabled) {
|
||||
if (!o.reply && o.interactions?.length) {
|
||||
const choices = o.interactions.filter(card => card.kind === "ask_user_questions" && card.status === "pending" && /Cases/i.test(JSON.stringify(card.payload)));
|
||||
if (choices.length !== 1) throw new Error("Missing unambiguous Cases availability choice in real UI");
|
||||
const request = ctx.input.page.locator(`[id=${JSON.stringify(`interaction-${choices[0]!.id}`)}]`);
|
||||
// The person opens the saved choice from its timeline chip and sees that same card docked in the composer.
|
||||
// The question's wording is left to human review.
|
||||
const choices = pendingChoices(o);
|
||||
if (choices.length !== 1) throw new Error("Missing unambiguous pending choice card in real UI");
|
||||
const request = ctx.input.page.locator(unansweredQuestionSelector(choices[0]!.id));
|
||||
await expect(request).toBeVisible(); await request.click();
|
||||
await expect(ctx.input.page.getByText(/Cases (?:is |are )?(?:disabled|unavailable|not enabled|not available)/i).first()).toBeVisible();
|
||||
await expect(ctx.input.page.getByText(/ordinary task|regular task|normal task|enable Cases/i).first()).toBeVisible();
|
||||
await expect(ctx.input.page.getByTestId("task-chat-composer-takeover").locator(`[data-interaction-id=${JSON.stringify(choices[0]!.id)}]`)).toBeVisible();
|
||||
}
|
||||
const settings = experimentalSettingsHref(o);
|
||||
if (settings) {
|
||||
@@ -103,7 +111,7 @@ export const integratorCasesCases: readonly ChatCatalogCase[] = [
|
||||
if (!ctx.issue()) throw new Error("Customer bug context requires its actual standing conversation");
|
||||
await ctx.input.api.patch(`/api/issues/${ctx.issue()!.id}`, { description: "Customer bug: the checkout button stops responding on mobile Safari after a trial item and a test address are entered. The person is asking about this bug." });
|
||||
return {};
|
||||
}), oracleVersion: "paperclip.chat-catalog.cases-disabled.structural.v1" },
|
||||
}), oracleVersion: "paperclip.chat-catalog.cases-disabled.structural.v2" },
|
||||
{ ...define("cases.browse-cases", true, async ctx => {
|
||||
const launch = await project(ctx, "Launch"); const other = await project(ctx, "Research");
|
||||
const create = (title: string, projectId: string, status: string) => ctx.input.api.post<CaseRecord>(`${root(ctx)}/cases`, { caseType: "customer", key: `browse-${ctx.input.nonce}-${title.toLowerCase().replace(/[^a-z0-9]+/g, "-")}`, title, projectId, status });
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { savedDesignInstruction, withRenderedCommentReference, validRenderedCommentReference } from "./internal-comment-reference.js";
|
||||
import { linkedQuestionTakeover, observeLinkedTaskQuestion } from "../chat-linked-question.js";
|
||||
import { linkedQuestionTakeover, observeLinkedTaskQuestion, questionCardNamesCandidates, type ClarificationObservation } from "../chat-linked-question.js";
|
||||
import { withEditOwnership } from "./internal-edit-ownership.js";
|
||||
import { verifyDistinctTaskFixtures } from "./internal-distinct-fixtures.js";
|
||||
import { delegatedTaskCases } from "./internal-task-creation.js";
|
||||
@@ -34,10 +34,13 @@ export function grade_tasks(value: unknown) {
|
||||
case "tasks.ambiguous-task": {
|
||||
const other = entity(o.state, "otherTask");
|
||||
const first = observation(o.state.clarification);
|
||||
// The candidates are identified by rendered links to both tasks, or by the run's saved question card offering
|
||||
// each task as its own choice: card options render as plain-text choice buttons, which cannot hold links.
|
||||
const linked = !!first && !!before && hasLink(first, "issues", before) && hasLink(first, "issues", other);
|
||||
const offered = !!first && !!before && questionCardNamesCandidates((first as ClarificationObservation).questionCards ?? [], [before, other], first.runs[0]?.id, o.leadId);
|
||||
c.add("clarify-before-write", !!first && !!before && sameTask(before, first.tasks.find(row => row.id === before.id))
|
||||
&& sameTask(other, first.tasks.find(row => row.id === other.id))
|
||||
&& hasLink(first, "issues", before) && hasLink(first, "issues", other),
|
||||
"The first completed turn changes neither task and gives the person links to both candidates.");
|
||||
&& sameTask(other, first.tasks.find(row => row.id === other.id)) && (linked || offered),
|
||||
"The first completed turn changes neither task and identifies both candidates with links or as separate saved question choices.");
|
||||
c.add("only-selected-task-changed", !!before && current?.priority === "high" && sameTask(before, current, ["priority"])
|
||||
&& sameTask(other, o.tasks.find(row => row.id === other.id)) && o.tasks.length === 2,
|
||||
"After the explicit Launch selection, only that task's requested priority changes.");
|
||||
@@ -91,7 +94,7 @@ function ambiguousTask(): ChatCatalogCase {
|
||||
await verifyDistinctTaskFixtures(ctx, [t, otherTask]);
|
||||
return { project: p, task: t, otherProject, otherTask };
|
||||
});
|
||||
return { ...base, oracleVersion: "paperclip.chat-catalog.internal.ambiguous-task.v4", sourceFiles: [...base.sourceFiles, "chat-catalog-domains/internal-distinct-fixtures.ts", "chat-linked-question.ts", "chat-catalog-domains/identity-support.ts", "chat-rendered-links.ts"], expectedRunCount: 2, expectedHumanMessageCount: 2, attemptTimeoutMs: 420_000,
|
||||
return { ...base, oracleVersion: "paperclip.chat-catalog.internal.ambiguous-task.v5", sourceFiles: [...base.sourceFiles, "chat-catalog-domains/internal-distinct-fixtures.ts", "chat-linked-question.ts", "chat-catalog-domains/identity-support.ts", "chat-rendered-links.ts"], expectedRunCount: 2, expectedHumanMessageCount: 2, attemptTimeoutMs: 420_000,
|
||||
async act(ctx, state) {
|
||||
await pollUntil({ label: "Task ambiguity clarification", deadlineAt: Date.now() + 150_000,
|
||||
load: () => observeInternal(ctx, state), accept: o => o.runs.length === 1 && o.runs.every(row => row.status === "succeeded")
|
||||
|
||||
@@ -72,30 +72,30 @@ it("rejects an unbound later worker in the grade when its prompt lacks the direc
|
||||
const o = valid(); o.state.cancelDirectionRunBound = promptCarries(`{"body":${JSON.stringify("unrelated")}}`, "Remove the contact section.\nKeep fonts.");
|
||||
expect(gradeDiscard(o).passed).toBe(false);
|
||||
});
|
||||
const label = "78f7e3a6c6ec-1";
|
||||
const initialPlan = "# Website Plan (Initial)\n\n## Status\n\nInitial plan document created.\n\n## Accessible font guidance (retained)\n\n- Readable base size.\n";
|
||||
const consumedPlan = `# Website Plan\n\n## Human direction\n\n- Add a contact section to the saved website plan, labelled \`${label}\`.\n\n## Contact section (label: ${label})\n\n- **Purpose:** give visitors a clear way to reach the site owner.\n\n## Accessible font guidance (retained)\n\n- Readable base size.\n`;
|
||||
const revs = (finalBody: string, extraSave?: string) => [
|
||||
const label = "e972d9c1aa95-1";
|
||||
const initialPlan = "# Website Plan (Initial)\n\n## Accessible Font Guidance\n\n- Readable base size.\n\n## Proposed Pages\n1. Home\n2. Contact\n";
|
||||
// Full cohort 11: the consuming run saved the label as a plain line under a "## Contact" heading.
|
||||
const consumedPlan = `# Website Plan (Initial)\n\n## Accessible Font Guidance\n\n- Readable base size.\n\n## Contact\nSection label: \`${label}\`\n\n- Add a contact form.\n\n## Proposed Pages\n1. Home\n2. Contact\n`;
|
||||
const finalPlan = "# Website Plan (Initial)\n\n## Accessible Font Guidance\n\n- Readable base size.\n\n## Proposed Pages\n1. Home\n";
|
||||
const revs = (finalBody: string) => [
|
||||
{ id: "r1", revisionNumber: 1, createdByRunId: "initial", body: initialPlan },
|
||||
{ id: "r2", revisionNumber: 2, createdByRunId: "consumer", body: consumedPlan },
|
||||
...(extraSave ? [{ id: "r3", revisionNumber: 3, createdByRunId: "consumer", body: extraSave }] : []),
|
||||
{ id: "final", revisionNumber: 4, createdByRunId: "final", body: finalBody },
|
||||
{ id: "final", revisionNumber: 3, createdByRunId: "final", body: finalBody },
|
||||
];
|
||||
it("full cohort 6: the labelled heading is gone and a superseded note may still name the label", () => {
|
||||
const final = `# Website Plan\n\n## Human direction\n\n- No contact section is included; the earlier \`${label}\` contact-section request has been superseded.\n\n## Accessible font guidance (retained)\n\n- Readable base size.\n`;
|
||||
expect(documentContactRemoved(revs(final), "final", "consumer", label)).toBe(true);
|
||||
it("full cohort 11: passes when the final revision drops the setup label, whatever form the label was saved in", () => {
|
||||
expect(documentContactRemoved(revs(finalPlan), "final", label)).toBe(true);
|
||||
for (const form of [`## Contact section (label: ${label})`, `### Reach us (${label})`, `Contact - ${label}`]) {
|
||||
const consumed = consumedPlan.replace(`## Contact\nSection label: \`${label}\``, form);
|
||||
expect(documentContactRemoved([...revs(finalPlan)].map(row => row.id === "r2" ? { ...row, body: consumed } : row), "final", label), form).toBe(true);
|
||||
}
|
||||
});
|
||||
it("fails while a heading still carries the label, kept or renamed", () => {
|
||||
expect(documentContactRemoved(revs(consumedPlan), "final", "consumer", label)).toBe(false);
|
||||
expect(documentContactRemoved(revs(consumedPlan.replace(`## Contact section (label: ${label})`, `### Reach us (${label})`)), "final", "consumer", label)).toBe(false);
|
||||
it("fails while the final revision still carries the setup label in any form", () => {
|
||||
expect(documentContactRemoved(revs(consumedPlan), "final", label)).toBe(false);
|
||||
expect(documentContactRemoved(revs(`${finalPlan}\nContact (${label}): email us.\n`), "final", label)).toBe(false);
|
||||
});
|
||||
it("compares against the plan from before the consuming run first saved the label, across its repeated saves", () => {
|
||||
expect(documentContactRemoved(revs(initialPlan, `${consumedPlan}\n## Next steps\n\nConfirm contact details.\n`), "final", "consumer", label)).toBe(true);
|
||||
expect(documentContactRemoved(revs(consumedPlan, `${consumedPlan}\n## Next steps\n\nConfirm contact details.\n`), "final", "consumer", label)).toBe(false);
|
||||
});
|
||||
it("fails when no labelled section was ever saved, so removal cannot be shown", () => {
|
||||
const unlabelled = consumedPlan.replace(`## Contact section (label: ${label})`, "## Contact section");
|
||||
expect(documentContactRemoved([...revs(initialPlan)].map(row => row.id === "r2" ? { ...row, body: unlabelled } : row), "final", "consumer", label)).toBe(false);
|
||||
it("fails when the final revision is missing or the label is empty", () => {
|
||||
expect(documentContactRemoved(revs(finalPlan), "missing", label)).toBe(false);
|
||||
expect(documentContactRemoved(revs(finalPlan), "final", "")).toBe(false);
|
||||
});
|
||||
|
||||
function citedInTurn() {
|
||||
|
||||
@@ -22,16 +22,13 @@ export function promptCarries(prompt: string | undefined, body: string) {
|
||||
export function replyLinkHrefs(reply: string, rendered: readonly string[]) {
|
||||
return [...new Set([...rendered, ...[...reply.matchAll(/\]\(([^)\s]+)\)/g)].map(match => match[1]!)])];
|
||||
}
|
||||
const labelledHeading = (body: string, label: string) => body.split(/\r?\n/).some(line => /^#{1,6}\s/.test(line) && line.includes(label));
|
||||
export function documentContactRemoved(rows: ReadonlyArray<{ id: string; body?: string | null; createdByRunId?: string | null }>, latestRevisionId: unknown, consumerRunId: string, label: string) {
|
||||
const number = (row: object) => Number((row as { revisionNumber?: unknown }).revisionNumber);
|
||||
const revisions = [...rows].sort((a, b) => number(a) - number(b));
|
||||
const final = revisions.find(row => row.id === latestRevisionId);
|
||||
const saves = revisions.filter(row => row.createdByRunId === consumerRunId && (row.body ?? "").includes(label));
|
||||
if (!saves.length) return !(final?.body ?? "").includes(label);
|
||||
const previous = revisions.filter(row => number(row) < number(saves[0]!)).at(-1);
|
||||
const hadSection = [previous, ...saves].some(row => labelledHeading(row?.body ?? "", label));
|
||||
return !!final && hadSection && !labelledHeading(final.body ?? "", label);
|
||||
/**
|
||||
* The cancelled request's setup label must be absent from the document's final revision. Whether the label
|
||||
* was ever saved is checked separately on the consuming run's revisions; the plan's layout is not graded.
|
||||
*/
|
||||
export function documentContactRemoved(rows: ReadonlyArray<{ id: string; body?: string | null }>, latestRevisionId: unknown, label: string) {
|
||||
const final = rows.find(row => row.id === latestRevisionId);
|
||||
return !!final && label.length > 0 && !(final.body ?? "").includes(label);
|
||||
}
|
||||
export function gradeDiscard(value: unknown) {
|
||||
const c = checks(), o = runtimeObservation(value);
|
||||
@@ -68,7 +65,7 @@ export function gradeDiscard(value: unknown) {
|
||||
return c.result();
|
||||
}
|
||||
export const runtimeDiscardCases = [runtimeDefinition("steering.discard-queued-message", ctx => humanQueuedRuntime(ctx, contactText), gradeDiscard, {
|
||||
authentication: "human-session", expectedHumanMessageCount: 2, expectedRunCount: 5, minimumRunCount: 4, attemptTimeoutMs: 1_020_000, oracleVersion: "paperclip.chat-catalog.runtime-discard.v7",
|
||||
authentication: "human-session", expectedHumanMessageCount: 2, expectedRunCount: 5, minimumRunCount: 4, attemptTimeoutMs: 1_020_000, oracleVersion: "paperclip.chat-catalog.runtime-discard.v8",
|
||||
sourceFiles: ["chat-catalog-domains/runtime-discard.ts", "chat-catalog-domains/runtime-queue-controls.ts", "chat-catalog-domains/runtime-queue.ts", "chat-catalog-domains/runtime-cited-delivery.ts", "chat-catalog-domains/runtime-busy-worker.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/identity-support.ts", "chat-scenarios/steering.yaml"],
|
||||
setupPolicy: runtimeQueueCases[0]!.setupPolicy, verifySetup: runtimeQueueCases[0]!.verifySetup,
|
||||
async act(ctx, state) {
|
||||
@@ -152,7 +149,7 @@ export const runtimeDiscardCases = [runtimeDefinition("steering.discard-queued-m
|
||||
const latest = o.revisions.filter(row => row.createdByRunId === state.finalWorkerRunId && row.createdByAgentId === worker
|
||||
&& o.documents.some(doc => doc.id === row.documentId && doc.latestRevisionId === row.id));
|
||||
state.finalPlanWithoutContact = latest.length > 0 && o.documents.every(doc => documentContactRemoved(
|
||||
o.revisions.filter(row => row.documentId === doc.id), doc.latestRevisionId, String(state.consumerRunId), label));
|
||||
o.revisions.filter(row => row.documentId === doc.id), doc.latestRevisionId, label));
|
||||
return o;
|
||||
},
|
||||
async cleanup(ctx, state) { try { await cleanupIdentityProbes(ctx); } finally { await releaseRuntimeWorker(state); } },
|
||||
|
||||
@@ -82,7 +82,7 @@ export async function verifyRuntimeResult(ctx: ChatCatalogContext, state: Record
|
||||
}
|
||||
/**
|
||||
* A pending interaction renders inline in the thread as `#interaction-<id>`,
|
||||
* or docked in the composer takeover, which carries its title but no id.
|
||||
* or docked in the composer takeover, which carries its title and data-interaction-id but not the page anchor id.
|
||||
*/
|
||||
export function interactionSurface(page: Page, interaction: { id: string; title?: string | null }) {
|
||||
const inline = page.locator(`[id=${JSON.stringify(`interaction-${interaction.id}`)}]`);
|
||||
|
||||
@@ -42,3 +42,29 @@ it("grades the clarification by saved state and candidate links, not by its word
|
||||
const both = structuredClone(final); both.tasks[1]!.priority = "high";
|
||||
expect(grade_tasks(both).passed).toBe(false);
|
||||
});
|
||||
// Full cohort 11: the question card had no linked context; its options named each candidate with its path as
|
||||
// plain text. Choice options render inside radio buttons, which cannot hold links, so the saved options are graded.
|
||||
it("accepts a saved question card that offers each candidate as its own choice, and rejects one that does not", () => {
|
||||
const input = valid();
|
||||
const card = (options: Array<{ id: string; label: string; description?: string }>, sourceRunId = "run") => ({ ...structuredClone(input.interactions[0]!), sourceRunId,
|
||||
payload: { version: 1, questionSet: { schema: "paperclip.question_set.v1", questions: [{ id: "task_choice", prompt: "Which landing-page task should become high priority?", answerMode: "single_select", required: true, options }] } } });
|
||||
const retained = [{ id: "com1", label: "RUN-1 (Launch project)", description: "Landing page task RUN-1 in the Launch project: /RUN/issues/RUN-1" },
|
||||
{ id: "com2", label: "RUN-2 (Experiments project)", description: "Landing page task RUN-2 in the Experiments project: /RUN/issues/RUN-2" },
|
||||
{ id: "both", label: "Both", description: "Mark both landing-page tasks as high priority: /RUN/issues/RUN-1 and /RUN/issues/RUN-2" }];
|
||||
const graded = (questionCards: unknown[]) => {
|
||||
const first = { ...input.observation, reply: "", links: [], questionCards };
|
||||
return grade_tasks({ ...structuredClone(input.observation), state: { ...input.observation.state, clarification: first },
|
||||
tasks: [{ ...input.observation.tasks[0]!, priority: "high" }, structuredClone(input.observation.tasks[1]!)],
|
||||
links: ["/RUN/issues/RUN-1"], activity: [{ id: "update", action: "issue.updated", entityId: "launch-task", agentId: "ceo" }] }).passed;
|
||||
};
|
||||
expect(graded([card(retained)])).toBe(true);
|
||||
expect(graded([card([{ id: "a", label: "Launch", description: "The RUN-1 task" }, { id: "b", label: "Experiments", description: "The RUN-2 task" }])])).toBe(true);
|
||||
expect(graded([])).toBe(false);
|
||||
expect(graded([card([retained[0]!, retained[2]!])])).toBe(false);
|
||||
expect(graded([card([{ id: "a", label: "Launch" }, { id: "b", label: "Experiments" }])])).toBe(false);
|
||||
expect(graded([card([retained[0]!, { id: "c", label: "RUN-12 (Archive)" }])])).toBe(false);
|
||||
expect(graded([card(retained, "other-run")])).toBe(false);
|
||||
for (const withdrawn of [{ status: "cancelled" }, { status: "expired" }, { dismissedAt: "2026-10-28T12:00:45Z" }, { resolvedAt: "2026-10-28T12:00:45Z" }]) {
|
||||
expect(graded([{ ...card(retained), ...withdrawn }]), JSON.stringify(withdrawn)).toBe(false);
|
||||
}
|
||||
});
|
||||
@@ -78,7 +78,22 @@ export function readLinkedQuestionDom(input: NonNullable<ReturnType<typeof linke
|
||||
return { ...input, links: [...new Set(qualified.map(link => link.href))], text: prompts[0]!.textContent ?? "" };
|
||||
}
|
||||
export type LinkedQuestionEvidence = NonNullable<ReturnType<typeof readLinkedQuestionDom>>;
|
||||
export type ClarificationObservation = InternalObservation & { linkedQuestionEvidence?: LinkedQuestionEvidence };
|
||||
export type ClarificationObservation = InternalObservation & { linkedQuestionEvidence?: LinkedQuestionEvidence; questionCards?: AskUserQuestionsInteraction[] };
|
||||
/**
|
||||
* Whether the run's saved question card offers each candidate task as its own choice. Option text is plain
|
||||
* text inside a choice button, so it cannot carry links; a candidate counts when one option names its identifier
|
||||
* (the setup literal) and no other candidate's. The option wording is left to human review.
|
||||
*/
|
||||
export function questionCardNamesCandidates(cards: readonly AskUserQuestionsInteraction[], tasks: ReadonlyArray<{ id: string; identifier?: string | null }>, runId: string | undefined, leadId: string) {
|
||||
const names = (text: string, task: { identifier?: string | null }) => !!task.identifier
|
||||
&& new RegExp(`(?<![A-Za-z0-9-])${task.identifier.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}(?![A-Za-z0-9])`, "i").test(text);
|
||||
return !!runId && tasks.length > 1 && new Set(tasks.map(task => task.identifier)).size === tasks.length && cards.some(card => card.kind === "ask_user_questions"
|
||||
&& card.status === "pending" && !card.dismissedAt && !card.resolvedAt && card.createdByAgentId === leadId && card.sourceRunId === runId
|
||||
&& (card.payload.questionSet?.questions ?? card.payload.questions ?? []).some(question => tasks.every(task => (question.options ?? []).some(option => {
|
||||
const text = `${option.label ?? ""}\n${option.description ?? ""}`;
|
||||
return names(text, task) && tasks.every(other => other.id === task.id || !names(text, other));
|
||||
}))));
|
||||
}
|
||||
export async function observeLinkedTaskQuestion(ctx: ChatCatalogContext, state: Record<string, unknown>, observation: InternalObservation): Promise<ClarificationObservation> {
|
||||
const source = ctx.issue(), prefix = ctx.input.fixtures.company.issuePrefix;
|
||||
if (!source || !prefix || ctx.scenario.id !== "tasks.ambiguous-task") return observation;
|
||||
@@ -88,9 +103,10 @@ export async function observeLinkedTaskQuestion(ctx: ChatCatalogContext, state:
|
||||
]);
|
||||
const binding = linkedQuestionBinding({ origin: observation.origin, pageUrl: ctx.input.page.url(), companyPrefix: prefix, sourceId: source.id, leadId: observation.leadId,
|
||||
query: ctx.scenario.query, observation, comments, interactions, tasks: [state.task as Entity, state.otherTask as Entity] });
|
||||
if (!binding) return observation;
|
||||
const questionCards = interactions.filter(card => card.kind === "ask_user_questions" && card.issueId === source.id);
|
||||
if (!binding) return { ...observation, questionCards };
|
||||
const evidence = await ctx.input.page.evaluate(readLinkedQuestionDom, binding);
|
||||
return evidence ? { ...observation, reply: [observation.reply, evidence.text].filter(Boolean).join("\n"), links: [...new Set([...observation.links, ...evidence.links])], linkedQuestionEvidence: evidence } : observation;
|
||||
return evidence ? { ...observation, reply: [observation.reply, evidence.text].filter(Boolean).join("\n"), links: [...new Set([...observation.links, ...evidence.links])], linkedQuestionEvidence: evidence, questionCards } : { ...observation, questionCards };
|
||||
}
|
||||
export function linkedQuestionTakeover(page: Page, evidence: LinkedQuestionEvidence): Locator {
|
||||
return page.getByTestId("task-chat-thread").filter({ has: page.locator(`[id=${JSON.stringify(`comment-${evidence.anchorCommentId}`)}]`) })
|
||||
|
||||
@@ -11,18 +11,19 @@ vi.mock("./chat-catalog-domains/external-support.js", () => ({ definition: () =>
|
||||
import { secretProposalCardSelector } from "./chat-catalog-domains/external-secret-proposal.js";
|
||||
afterEach(() => { document.body.innerHTML = ""; });
|
||||
it("selects the actual expanded proposal card when the pending trigger has the same production interaction ID", () => {
|
||||
document.body.innerHTML = `<button id="interaction-owned" data-testid="task-chat-pending-request" aria-label="Open request: Confirm secret binding">Confirm secret binding</button>
|
||||
<div id="interaction-owned" data-testid="task-chat-interaction"><p>Deployment publishing key</p><p>Morgan</p><p>env.PUBLISHING_API_KEY</p></div>
|
||||
<div id="interaction-other" data-testid="task-chat-interaction"><p>Other request</p></div>`;
|
||||
expect(document.querySelectorAll('[id="interaction-owned"]')).toHaveLength(2);
|
||||
// The timeline trigger owns the page anchor id; the docked card carries only the interaction data attribute.
|
||||
document.body.innerHTML = `<button id="interaction-owned" data-interaction-id="owned" data-testid="task-chat-pending-request" aria-label="Open request: Confirm secret binding">Confirm secret binding</button>
|
||||
<div data-interaction-id="owned" data-testid="task-chat-interaction"><p>Deployment publishing key</p><p>Morgan</p><p>env.PUBLISHING_API_KEY</p></div>
|
||||
<div data-interaction-id="other" data-testid="task-chat-interaction"><p>Other request</p></div>`;
|
||||
expect(document.querySelectorAll('[data-interaction-id="owned"]')).toHaveLength(2);
|
||||
const selected = document.querySelectorAll(secretProposalCardSelector("owned"));
|
||||
expect(selected).toHaveLength(1); expect(selected[0]?.tagName).toBe("DIV");
|
||||
for (const fact of ["Deployment publishing key", "Morgan", "env.PUBLISHING_API_KEY"]) expect(selected[0]?.textContent).toContain(fact);
|
||||
expect(document.querySelectorAll(secretProposalCardSelector("missing"))).toHaveLength(0);
|
||||
});
|
||||
it("a trigger, unrelated card or completed receipt cannot stand in for the bound proposal card", () => {
|
||||
document.body.innerHTML = `<button id="interaction-owned" data-testid="task-chat-pending-request">Confirm secret binding</button>
|
||||
<div id="interaction-foreign" data-testid="task-chat-interaction">Deployment publishing key Morgan env.PUBLISHING_API_KEY</div>
|
||||
<div id="interaction-owned" data-testid="task-chat-interaction-receipt">Approved</div>`;
|
||||
document.body.innerHTML = `<button id="interaction-owned" data-interaction-id="owned" data-testid="task-chat-pending-request">Confirm secret binding</button>
|
||||
<div data-interaction-id="foreign" data-testid="task-chat-interaction">Deployment publishing key Morgan env.PUBLISHING_API_KEY</div>
|
||||
<div data-interaction-id="owned" data-testid="task-chat-interaction-receipt">Approved</div>`;
|
||||
expect(document.querySelectorAll(secretProposalCardSelector("owned"))).toHaveLength(0);
|
||||
});
|
||||
@@ -24,10 +24,10 @@ async function fixture(request: APIRequestContext) {
|
||||
res.writeHead(200, { "content-type": "application/json" }); res.end(JSON.stringify(card));
|
||||
} else {
|
||||
res.writeHead(200, { "content-type": "text/html" });
|
||||
res.end(`<section data-testid="task-chat-composer-takeover"><div id="interaction-current">
|
||||
res.end(`<section data-testid="task-chat-composer-takeover"><div data-interaction-id="current">
|
||||
<p>Approve revised plan</p><button onclick="fetch('/api/issues/parent/interactions/current/accept',{method:'POST'})">Confirm</button>
|
||||
<button>Dismiss request</button></div></section>
|
||||
<div id="interaction-stale"><button onclick="fetch('/api/issues/parent/interactions/stale/accept',{method:'POST'})">Confirm</button></div>`);
|
||||
<div id="interaction-stale" data-interaction-id="stale"><button onclick="fetch('/api/issues/parent/interactions/stale/accept',{method:'POST'})">Confirm</button></div>`);
|
||||
}
|
||||
});
|
||||
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
|
||||
|
||||
@@ -64,7 +64,7 @@ export async function confirmContinuationPlan(input: {
|
||||
expect(current.id, "the reviewed confirmation must still be current").toBe(expected.id);
|
||||
expect(sameTarget(current.payload.target, expected.payload.target), "the reviewed plan must not change before approval").toBe(true);
|
||||
const card = input.page.getByTestId("task-chat-composer-takeover")
|
||||
.locator(`[id=${JSON.stringify(`interaction-${current.id}`)}]`);
|
||||
.locator(`[data-interaction-id=${JSON.stringify(current.id)}]`);
|
||||
await expect(card, "the exact current confirmation must be visible").toBeVisible();
|
||||
await card.getByRole("button", { name: current.payload.acceptLabel ?? "Approve", exact: true }).click();
|
||||
const saved = await pollUntil({
|
||||
|
||||
@@ -2049,7 +2049,7 @@ for (const execution of executions) {
|
||||
await expect(turnReply).toBeVisible({ timeout: reviewUiTimeout() });
|
||||
const pendingReview = waitingState.interactions.find(isPendingWarmConfirmation)!;
|
||||
const reviewCard = page.getByTestId("task-chat-interaction")
|
||||
.and(page.locator(`[id=${JSON.stringify(`interaction-${pendingReview.id}`)}]`));
|
||||
.and(page.locator(`[data-interaction-id=${JSON.stringify(pendingReview.id)}]`));
|
||||
await expect(reviewCard).toBeVisible({ timeout: reviewUiTimeout() });
|
||||
await expect(reviewCard.getByRole("button", { name: "Continue work", exact: true }))
|
||||
.toBeEnabled({ timeout: reviewUiTimeout() });
|
||||
|
||||
@@ -262,6 +262,32 @@ describe("TaskChatInteractionCard", () => {
|
||||
expect(preview).not.toMatch(/[`*_~[\]]|(^|\s)- /);
|
||||
});
|
||||
|
||||
// Full cohort 11: the unanswered-question chip and the composer takeover showed the same
|
||||
// pending question at once, both with id "interaction-<id>", so the page had a duplicate id.
|
||||
it("keeps the interaction id unique while its timeline chip and takeover card are both shown", () => {
|
||||
const interaction = pendingAskUserQuestionsInteraction;
|
||||
flushSync(() => {
|
||||
root.render(
|
||||
<TooltipProvider>
|
||||
<ThemeProvider>
|
||||
<TaskChatInteractionCard item={interactionItem(interaction)} showUnansweredQuestion onReviewRequest={() => {}} />
|
||||
<div data-testid="task-chat-composer-takeover">
|
||||
<TaskChatInteractionCard item={interactionItem(interaction)} presentation="takeover" />
|
||||
</div>
|
||||
</ThemeProvider>
|
||||
</TooltipProvider>,
|
||||
);
|
||||
});
|
||||
const anchors = container.querySelectorAll(`[id="interaction-${interaction.id}"]`);
|
||||
expect(anchors).toHaveLength(1);
|
||||
expect(anchors[0]!.getAttribute("data-testid")).toBe("task-chat-unanswered-question");
|
||||
const takeover = container.querySelector(`[data-testid="task-chat-composer-takeover"] [data-interaction-id="${interaction.id}"]`);
|
||||
expect(takeover).not.toBeNull();
|
||||
expect(takeover!.id).toBe("");
|
||||
const ids = [...container.querySelectorAll("[id]")].map(node => node.id);
|
||||
expect(new Set(ids).size).toBe(ids.length);
|
||||
});
|
||||
|
||||
it("renders a plan confirmation as a linked document preview", () => {
|
||||
const interaction = createRequestConfirmation({
|
||||
sourceRunId: "run-plan",
|
||||
|
||||
@@ -50,8 +50,12 @@ export function TaskChatInteractionCard({
|
||||
}: TaskChatInteractionCardProps) {
|
||||
const { enabled: coordinatorEnabled } = useCoordinatorWorkspaceEnabled();
|
||||
const interaction = item.interaction;
|
||||
// The timeline item owns the interaction's page anchor. The composer takeover
|
||||
// shows the same interaction at the same time, so it carries only the data
|
||||
// attribute: a DOM id must stay unique.
|
||||
const anchorId = presentation === "takeover" ? undefined : `interaction-${interaction.id}`;
|
||||
if (coordinatorEnabled && interaction.kind === "suggest_tasks" && !shouldHideInteractionCard(interaction)) {
|
||||
return <div id={`interaction-${interaction.id}`} data-testid="task-chat-interaction" className="w-full">
|
||||
return <div id={anchorId} data-interaction-id={interaction.id} data-testid="task-chat-interaction" className="w-full">
|
||||
<TaskChatCompactInteractionCard interaction={interaction} presentation={presentation} draftKey={draftKey} {...cardProps} />
|
||||
</div>;
|
||||
}
|
||||
@@ -60,7 +64,7 @@ export function TaskChatInteractionCard({
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
id={`interaction-${interaction.id}`}
|
||||
id={anchorId} data-interaction-id={interaction.id}
|
||||
data-testid="task-chat-unanswered-question"
|
||||
aria-label={`Answer question: ${prompt}`}
|
||||
disabled={!onReviewRequest}
|
||||
@@ -79,7 +83,7 @@ export function TaskChatInteractionCard({
|
||||
return <IssueThreadInteractionCard interaction={interaction} {...cardProps} />;
|
||||
}
|
||||
return (
|
||||
<div id={`interaction-${interaction.id}`} data-testid="task-chat-tool-review" className="w-full space-y-2">
|
||||
<div id={anchorId} data-interaction-id={interaction.id} data-testid="task-chat-tool-review" className="w-full space-y-2">
|
||||
{interaction.status === "pending" ? (
|
||||
<div className="flex flex-wrap items-center gap-2 text-sm">
|
||||
<AppLogo name={action.appDisplayName || action.toolDisplayName} size={36} />
|
||||
@@ -108,7 +112,7 @@ export function TaskChatInteractionCard({
|
||||
return (
|
||||
<div className="w-full space-y-2">
|
||||
{showPlan ? <TaskChatPlanPreviewCard source={{ kind: "saved", document: planDocument }} testId="plan-review-preview" /> : null}
|
||||
<button type="button" id={`interaction-${interaction.id}`} data-testid="task-chat-pending-request"
|
||||
<button type="button" id={anchorId} data-interaction-id={interaction.id} data-testid="task-chat-pending-request"
|
||||
aria-label={`Open request: ${label}`} disabled={!onReviewRequest}
|
||||
onClick={() => onReviewRequest?.(interaction.id)}
|
||||
className="group flex w-full items-center gap-2 rounded-sm px-1 py-1.5 text-left text-sm text-muted-foreground transition-colors hover:text-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring">
|
||||
@@ -140,7 +144,7 @@ export function TaskChatInteractionCard({
|
||||
if (!isPlanReview || !showPlanPreview) return null;
|
||||
return (
|
||||
<div
|
||||
id={`interaction-${interaction.id}`}
|
||||
id={anchorId} data-interaction-id={interaction.id}
|
||||
data-testid="task-chat-interaction"
|
||||
>
|
||||
<TaskChatPlanPreviewCard
|
||||
@@ -169,7 +173,7 @@ export function TaskChatInteractionCard({
|
||||
// card while it is current, and the receipt records who decided and why.
|
||||
return (
|
||||
<div
|
||||
id={`interaction-${interaction.id}`}
|
||||
id={anchorId} data-interaction-id={interaction.id}
|
||||
className="w-full space-y-2"
|
||||
data-testid="task-chat-interaction"
|
||||
>
|
||||
@@ -193,7 +197,7 @@ export function TaskChatInteractionCard({
|
||||
if (presentation === "timeline" && interaction.status !== "pending") {
|
||||
return (
|
||||
<div
|
||||
id={`interaction-${interaction.id}`}
|
||||
id={anchorId} data-interaction-id={interaction.id}
|
||||
className="tc-enter-bubble w-full"
|
||||
data-testid="task-chat-interaction"
|
||||
>
|
||||
@@ -210,7 +214,7 @@ export function TaskChatInteractionCard({
|
||||
}
|
||||
return (
|
||||
<div
|
||||
id={`interaction-${interaction.id}`}
|
||||
id={anchorId} data-interaction-id={interaction.id}
|
||||
className="tc-enter-bubble w-full"
|
||||
data-testid="task-chat-interaction"
|
||||
>
|
||||
|
||||
Reference in new issue
Block a user