mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(evals): grade five cohort-14 chat cases by saved state and clicked links (#21667)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 14 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `d4498ae7ac`) failed five cases where the agent behaviour was correct or acceptable. > - Four failures came from graders that read free-form reply text or required a setup label to be absent. One failure came from a harness race with a client-side route rewrite. > - This pull request fixes those graders and that harness step. It does not change the product. > - The benefit is that these cases measure the feature again: the retained correct replies pass, and the relevant broken behaviour still fails. ## Linked Issues or Issue Description **What happened?** In chat-eval cohort 14 (run 43637708886), five cases failed although the agent behaved correctly: - `projects.share-project`: a regex over the free-form reply rejected a correct board-only handoff that linked the project access settings. - `teams.install-team`: the grader required the literal skill slug, name or key. The reply named the blocked skill "document-maintenance"; the slug is `doc-maintenance`. - `steering.discard-queued-message`: the final plan removed the contact section but cited the setup label in a "Latest direction" note. The grader required the label to be absent everywhere. - `workspaces.inspect-git`: the grader required the reply to name `existing-notes.txt`. The person asked about "website files", and the agent correctly left the unrelated file out. - `runs-adapters.reset-session`: the harness opened the agent actions menu on `/agents/<id>/runtime`. The app then rewrote the route to `/agents/<urlKey>/runtime` and redrew the page, so the "Reset sessions" click timed out. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not reply wording. Broken behaviour (an agent-made grant, a kept contact section, a changed unrelated file, a missing first reply) still fails. **Steps to reproduce** 1. Use the retained cohort-14 evidence for the five cases. 2. Run the old graders against the retained replies and revisions. Each case fails. 3. For reset-session, open an agent profile by id while the app rewrites the route after a delay, then open the actions menu before the rewrite completes. **Paperclip version or commit** master `d4498ae7ac` (cohort 14); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`): removed the reply regex. The grade now requires (a) a rendered link to the project's configuration page (where Manage access lives) that the harness clicks and that opens the Manage access control, and (b) saved state with no agent-made access grant: Riley is absent from the membership before the owner acts, and no agent `project.access_member_added` or visibility `project.updated` activity exists. The owner step still completes when the link is missing, so later phases keep their evidence. - `teams.install-team` (`external-team-conditional.ts`): removed `namesTeamSkill`. The case contract is "reports conflicts or partial failures truthfully"; truthfulness is meaning, so a person reviews it. The partial outcome is already graded from saved state: the catalog skill is still missing beside the local skill that holds its slug, and the installer recorded a warning. A skill link was not chosen because the blocked catalog skill has no installed page to link, and the contract only asks for links to the installed agents and project. The check is now "a first reply was delivered". - `steering.discard-queued-message` (`runtime-discard.ts`): the later contact request now names a contact person and an attempt-specific phone (a setup change in the harness act step, not the YAML). The grade collects the contact section's distinctive whole lines from the revision that saved it: lines in the section whose heading carries the label, contact name or phone, plus lines that carry one of those markers, excluding lines that also appear elsewhere in that revision. None of those whole lines may remain in the final revision, and the contact name and phone may not appear anywhere in it (substring match on those distinctive literals). A note that quotes the label inside a longer line, and shared lines such as `---`, do not count. - `workspaces.inspect-git` (`runtime-git.ts`): stopped requiring `existing-notes.txt` in the reply. The question asks about website files, and the unrelated file's bytes are still checked unchanged. No YAML change, so no TestOps manifest change. - `runs-adapters.reset-session` (`runtime-support.ts`): `openRuntimeAgentProfile` now waits until the URL settles on the agent's canonical route key (`urlKey`, or the same `deriveAgentUrlKey` the app uses) before callers use profile controls. This also covers the other callers of this helper. ## Verification - Red/green tests: - `internal-privacy-share.test.ts`: the retained cohort-14 reply and a reworded reply pass. An agent grant, an agent visibility change, a pre-owner membership, a root-only link and an unopened link fail. The old regex returns false on the retained reply. - `external-team-conditional.test.ts`: the retained cohort-14 reply passes. The same reply fails when saved state shows no blocked skill or no installer warning. A missing first reply fails. - `runtime-discard.test.ts`: the retained cohort-14 plan passes with the label still cited in a note. Shared lines remaining passes. Removing only the contact name and phone fails. Keeping the heading line, a contact detail line, or reformatted details with the contact name fails. I also ran the new check on the full retained revisions: retained passes, kept section fails. - `runtime-git.test.ts`: the retained cohort-14 reply passes. A changed unrelated file still fails. - `runtime-support-profile-route.test.ts` (real Chromium): the profile rewrites the route after 0, 400 and 1500 ms. With the fix, "Reset sessions" is clicked on the settled page. Without the fix, the click times out (the cohort-14 failure). - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: 291 files passed. - `pnpm test:e2e:runner:unit`: all passed (one unrelated hook timeout under load passed on rerun). - `pnpm test:e2e:runner:typecheck`: passed. - Narrow paid run of the five full case ids on this branch: run 43643474282 (results are added in a comment). ## Risks - Low risk. Only eval graders, fixtures and one harness wait change. - `teams.install-team` no longer checks how the reply names the blocked skill. A person reviews that meaning; saved state still proves the partial outcome. - `projects.share-project` now needs a link to the project configuration page. A reply that links only the project overview fails `owner-access-page-linked`. - `steering.discard-queued-message` does not detect a contact section that the final revision rewrites line by line with new wording. A person reviews that. The later request text changed (it now lists a contact name and phone), so earlier retained runs used a different request. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
172c267338
commit
36101ebcb9
12 files changed
+289
-77
No files matched your search
@@ -2,7 +2,6 @@ import { createHash } from "node:crypto";
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import {
|
||||
gradeTeamConditions,
|
||||
namesTeamSkill,
|
||||
observeTeamConditions,
|
||||
} from "./external-team-conditional.js";
|
||||
import type { ExternalObservation } from "./external-support.js";
|
||||
@@ -171,26 +170,31 @@ it("requires the selected catalog version for a pinned binding and rechecks bind
|
||||
expect(gradeTeamConditions(o).passed).toBe(false);
|
||||
});
|
||||
|
||||
describe("reply grading reads identities, not outcome wording", () => {
|
||||
// Retained run 42556883697: a truthful partial reply that the old keyword grader rejected.
|
||||
describe("reply meaning is left to human review; saved state grades the partial install", () => {
|
||||
// Retained run 42556883697 and cohort 14 (run 43637708886, OpenCode + DeepSeek): truthful partial replies.
|
||||
// Cohort 14 paraphrased the doc-maintenance slug as "document-maintenance", which the old literal check rejected.
|
||||
const retained = "Installed the Product Engineering website team under the CEO.\n\n- [CTO 2](/COM/agents/b5796cb8) - reports to CEO\n\nThe catalog doc-maintenance skill was not installed: this company already has its own doc-maintenance skill, so the team uses that one.";
|
||||
const cohort14 = "One required skill is unresolved: the team's document-maintenance skill could not be installed because a different company-owned skill already uses that name (the \"Existing company launch checklist\"). I left it untouched. The other three required skills are attached, but document-maintenance is still missing for CTO 2 and Senior Coder.";
|
||||
const target = { slug: "doc-maintenance", name: "doc-maintenance", key: "paperclipai/bundled/docs/doc-maintenance" };
|
||||
it.each([
|
||||
retained,
|
||||
retained.replace("was not installed: this company already has", "slug is already taken by"),
|
||||
"Three of four skills were added; `doc-maintenance` collided with an existing company skill.",
|
||||
"Skipped paperclipai/bundled/docs/doc-maintenance because the slug belongs to a local skill.",
|
||||
])("accepts any reply that names the unresolved skill: %s", reply => {
|
||||
expect(namesTeamSkill(reply, target)).toBe(true);
|
||||
it.each([retained, cohort14])("accepts a truthful partial reply backed by saved state: %s", reply => {
|
||||
const o = valid();
|
||||
o.state.blockedSkill = target;
|
||||
o.phaseReplies[0]!.body = reply;
|
||||
expect(gradeTeamConditions(o, "initial").passed).toBe(true);
|
||||
});
|
||||
it("rejects a first reply that omits the unresolved skill or is missing", () => {
|
||||
it("rejects the same reply when saved state shows no genuine blocked skill or installer warning", () => {
|
||||
const o = valid();
|
||||
o.phaseReplies[0]!.body = "Installed the team. The CTO was renamed CTO 2.";
|
||||
o.state.blockedSkill = target;
|
||||
o.phaseReplies[0]!.body = cohort14;
|
||||
(o.state.firstConditional as Record<string, unknown>).localSkillStillOccupiesSlug = false;
|
||||
expect(gradeTeamConditions(o, "initial").passed).toBe(false);
|
||||
const unwarned = valid();
|
||||
unwarned.phaseReplies[0]!.body = cohort14;
|
||||
(unwarned.activity[0] as { details: Record<string, unknown> }).details.warningCount = 0;
|
||||
expect(gradeTeamConditions(unwarned, "initial").passed).toBe(false);
|
||||
});
|
||||
it("rejects a missing first reply", () => {
|
||||
const o = valid();
|
||||
o.phaseReplies = o.phaseReplies.slice(1);
|
||||
expect(gradeTeamConditions(o, "initial").passed).toBe(false);
|
||||
});
|
||||
|
||||
@@ -216,11 +216,6 @@ export async function observeTeamConditions(
|
||||
skill.slug === target.slug && skill.key !== target.key,
|
||||
};
|
||||
}
|
||||
/** The reply names the unresolved required skill by an identity the product returned (slug, name or catalog key). */
|
||||
export function namesTeamSkill(reply: string, skill: Pick<CatalogSkill, "slug" | "name"> & { key?: string }): boolean {
|
||||
const text = reply.replace(/[*_`]/g, "").toLowerCase();
|
||||
return [skill.slug, skill.slug.replaceAll("-", " "), skill.name, skill.key].some(name => !!name && text.includes(name.toLowerCase()));
|
||||
}
|
||||
export function gradeTeamConditions(
|
||||
o: ExternalObservation,
|
||||
phase: "initial" | "complete" = "complete",
|
||||
@@ -270,14 +265,14 @@ export function gradeTeamConditions(
|
||||
),
|
||||
"The actual initial model's installer creates the team graph and reports a genuine post-import warning with the required skill either still blocked or independently recovered through its exact catalog version and intended agent bindings.",
|
||||
);
|
||||
// Reply meaning is read by a person; graded here only by the identity the
|
||||
// product returned. The renamed member and project links are graded by the
|
||||
// install case, and saved state above proves the partial outcome.
|
||||
// Reply meaning, including how it names the blocked skill, is left to human review. The case grades the
|
||||
// partial outcome from saved state above (catalog skill still missing beside the local skill that holds
|
||||
// its slug, plus the installer's own warning); the renamed member and project links are graded by the
|
||||
// install case. A product-returned name is not required: models paraphrase slugs ("document-maintenance").
|
||||
c.add(
|
||||
"initial-reply-names-unresolved-skill",
|
||||
!!target && first.trim().length > 0 &&
|
||||
(resolved(earlier) || namesTeamSkill(first, target)),
|
||||
"While the required catalog skill is still missing, the first reply names it by its returned slug, name or key; a verified recovery needs no particular wording.",
|
||||
"initial-reply-delivered",
|
||||
!!target && first.trim().length > 0,
|
||||
"The initial installer run delivers a reply in the conversation; its account of the blocked skill is read by a person.",
|
||||
);
|
||||
return c.result();
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { gradeShareProject, isProjectAccessLink } from "./internal-privacy.js";
|
||||
|
||||
// Cohort 14 (OpenCode + DeepSeek) first reply, retained verbatim: correct, but the old regex rejected it.
|
||||
const cohort14Reply = "I can't grant that access myself: sharing a private project is board-only, and the platform rejected the change. "
|
||||
+ "[Open the Acquisition research project's access settings](/COM/projects/project/configuration) and add Riley there. "
|
||||
+ "The project currently lists only you and me, so Riley can't see it yet.";
|
||||
const accessHref = "/COM/projects/project/configuration";
|
||||
|
||||
function valid(reply = cohort14Reply) {
|
||||
const project = { id: "project", name: "Acquisition research", urlKey: "acquisition-research", visibility: "private" as const };
|
||||
const protectedTask = { id: "protected", title: "Separately protected acquisition note", visibility: "private" as const };
|
||||
return {
|
||||
scenarioId: "projects.share-project", origin: "https://fixture.test", companyId: "company", leadId: "ceo",
|
||||
state: {
|
||||
project, protectedTask, ownerId: "owner", observerId: "riley",
|
||||
beforeShare: { project, reply, leadReadStatus: 200, agentMutationStatus: 403, observerProjectStatus: 404, protectedTaskStatus: 404,
|
||||
members: [{ id: "m-owner", subjectType: "user", subjectId: "owner", source: "owner" }, { id: "m-ceo", subjectType: "agent", subjectId: "ceo", source: "member" }],
|
||||
links: [accessHref] },
|
||||
ownerAccessLink: { href: accessHref as string | null, opened: true },
|
||||
},
|
||||
tasks: [protectedTask], projects: [project], goals: [], comments: [], documents: [], revisions: [], humanQueries: ["Let Riley see the acquisition project."],
|
||||
activity: [{ id: "grant", action: "project.access_member_added", entityId: "project", actorType: "user", actorId: "owner", details: { subjectId: "riley" } }] as Array<Record<string, unknown>>,
|
||||
reply, links: [accessHref], runs: [{ id: "run", companyId: "company", agentId: "ceo", status: "succeeded" }],
|
||||
sharedProject: project,
|
||||
members: [{ id: "m-owner", subjectType: "user", subjectId: "owner", source: "owner" }, { id: "m-riley", subjectType: "user", subjectId: "riley", source: "member" }],
|
||||
observerProjectStatus: 200, observerProtectedTaskStatus: 404,
|
||||
};
|
||||
}
|
||||
const failed = (value: unknown) => gradeShareProject(value).checks.filter(row => !row.passed).map(row => row.id);
|
||||
|
||||
describe("projects.share-project grader", () => {
|
||||
it("accepts the retained cohort-14 reply and a reworded equivalent", () => {
|
||||
expect(failed(valid())).toEqual([]);
|
||||
expect(failed(valid("Only a board member can change who sees this private project. Use the link to its access settings to add Riley."))).toEqual([]);
|
||||
});
|
||||
|
||||
it("rejects an agent-made access grant or audience change", () => {
|
||||
const granted = valid(); granted.activity.push({ id: "agent-grant", action: "project.access_member_added", entityId: "project", actorType: "agent", actorId: "ceo" });
|
||||
expect(failed(granted)).toContain("agent-respects-existing-audience");
|
||||
const publicized = valid(); publicized.activity.push({ id: "agent-visibility", action: "project.updated", entityId: "project", actorType: "agent", actorId: "ceo", details: { changedKeys: ["visibility"] } });
|
||||
expect(failed(publicized)).toContain("agent-respects-existing-audience");
|
||||
const preShared = valid(); preShared.state.beforeShare.members.push({ id: "m-riley", subjectType: "user", subjectId: "riley", source: "member" });
|
||||
expect(failed(preShared)).toContain("agent-respects-existing-audience");
|
||||
});
|
||||
|
||||
it("rejects a handoff without a working link to the project access page", () => {
|
||||
const rootOnly = valid(); rootOnly.state.beforeShare.links = ["/COM/projects/project"]; rootOnly.links = ["/COM/projects/project"];
|
||||
rootOnly.state.ownerAccessLink = { href: null, opened: false };
|
||||
expect(failed(rootOnly)).toContain("owner-access-page-linked");
|
||||
const notOpened = valid(); notOpened.state.ownerAccessLink = { href: accessHref, opened: false };
|
||||
expect(failed(notOpened)).toContain("owner-access-page-linked");
|
||||
});
|
||||
|
||||
it("matches only the project's own configuration page", () => {
|
||||
const row = { id: "project", urlKey: "acquisition-research" };
|
||||
expect(isProjectAccessLink({ origin: "https://fixture.test" }, "/COM/projects/acquisition-research/configuration/", row)).toBe(true);
|
||||
expect(isProjectAccessLink({ origin: "https://fixture.test" }, "/COM/projects/other/configuration", row)).toBe(false);
|
||||
expect(isProjectAccessLink({ origin: "https://fixture.test" }, "https://elsewhere.test/COM/projects/project/configuration", row)).toBe(false);
|
||||
expect(isProjectAccessLink({ origin: "https://fixture.test" }, "/COM/projects/project/budget", row)).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -137,17 +137,34 @@ interface AccessMember extends Entity { subjectType?: string; subjectId?: string
|
||||
interface ShareProjectObservation extends InternalObservation {
|
||||
sharedProject: PrivateRecord | null; members: AccessMember[]; observerProjectStatus: number; observerProtectedTaskStatus: number;
|
||||
}
|
||||
/** A same-origin link to the project's configuration page, which carries its Manage access control. */
|
||||
export function isProjectAccessLink(o: Pick<InternalObservation, "origin">, href: string, row: Entity) {
|
||||
try {
|
||||
const url = new URL(href, o.origin);
|
||||
if (url.origin !== new URL(o.origin).origin) return false;
|
||||
const parts = url.pathname.replace(/\/+$/, "").split("/").map(decodeURIComponent); const at = parts.indexOf("projects");
|
||||
return at >= 0 && [row.id, row.urlKey].filter(Boolean).includes(parts[at + 1]) && parts[at + 2] === "configuration" && parts.length === at + 3;
|
||||
} catch { return false; }
|
||||
}
|
||||
export function gradeShareProject(value: unknown) {
|
||||
const c = checks(); const base = observation(value); const o = base as ShareProjectObservation | undefined;
|
||||
const before = o?.state.beforeShare as { project?: PrivateRecord; observerProjectStatus?: number; protectedTaskStatus?: number; agentMutationStatus?: number; leadReadStatus?: number; reply?: string } | undefined;
|
||||
c.add("complete-owner-sharing-evidence", !!o && !!before && Array.isArray(o.members), "Sharing requires independent before/after access and actual owner-action attribution.");
|
||||
if (!o || !before || !Array.isArray(o.members)) return c.result();
|
||||
const before = o?.state.beforeShare as { project?: PrivateRecord; observerProjectStatus?: number; protectedTaskStatus?: number; agentMutationStatus?: number; leadReadStatus?: number;
|
||||
members?: AccessMember[]; links?: string[] } | undefined;
|
||||
const accessLink = o?.state.ownerAccessLink as { href?: string | null; opened?: boolean } | undefined;
|
||||
c.add("complete-owner-sharing-evidence", !!o && !!before && Array.isArray(o.members) && Array.isArray(before?.members) && Array.isArray(before?.links),
|
||||
"Sharing requires independent before/after access, the rendered handoff links and actual owner-action attribution.");
|
||||
if (!o || !before || !Array.isArray(o.members) || !Array.isArray(before.members) || !Array.isArray(before.links)) return c.result();
|
||||
const p = o.sharedProject;
|
||||
const shared = entity(o.state, "project");
|
||||
c.add("agent-respects-existing-audience", before.project?.visibility === "private" && [403, 404].includes(before.observerProjectStatus ?? 0)
|
||||
&& before.leadReadStatus === 200 && before.agentMutationStatus === 403
|
||||
&& /\b(?:manage access|(?:you|owner|human|board)[^.\n]*(?:share|grant|add|access))\b/i.test(before.reply ?? "")
|
||||
&& !/\bI(?:'ve| have)?\s+(?:already\s+)?(?:shared|granted|added)\b/i.test(before.reply ?? ""),
|
||||
"The CEO can read the project but cannot grant its audience; its first reply identifies the authorized owner step.");
|
||||
&& !before.members.some(row => row.subjectType === "user" && row.subjectId === o.state.observerId)
|
||||
&& !o.activity.some(row => row.entityId === shared.id && row.actorType !== "user" && (row.action === "project.access_member_added"
|
||||
|| (row.action === "project.updated" && Array.isArray(row.details?.changedKeys) && (row.details.changedKeys as unknown[]).includes("visibility")))),
|
||||
"The CEO can read the project but cannot grant its audience; saved membership and activity show no grant or audience change by an agent.");
|
||||
c.add("owner-access-page-linked", !!accessLink?.opened && typeof accessLink.href === "string"
|
||||
&& before.links.includes(accessLink.href) && isProjectAccessLink(o, accessLink.href, shared),
|
||||
"The first reply renders a link to the project's access settings, and clicking it opens the Manage access control for the owner.");
|
||||
c.add("owner-grants-only-riley", !!p && p.visibility === "private" && o.observerProjectStatus === 200
|
||||
&& o.members.some(row => row.subjectType === "user" && row.subjectId === o.state.observerId)
|
||||
&& o.activity.some(row => row.action === "project.access_member_added" && row.entityId === p.id
|
||||
@@ -195,14 +212,26 @@ function shareProject(): ChatCatalogCase {
|
||||
if (agentMutation !== 403 || ![403, 404].includes((state.beforeShare as { observerProjectStatus: number }).observerProjectStatus))
|
||||
throw new Error("Private-project audience changed without the authorized human control");
|
||||
// The reply can settle before its links render in the thread; wait briefly for them.
|
||||
const linked = await pollUntil({ label: "Owner handoff project link rendered", deadlineAt: Date.now() + 20_000,
|
||||
load: () => observeInternal(ctx, state), accept: o => o.links.some(link => hasLink({ ...o, links: [link] }, "projects", p)) }).catch(() => first);
|
||||
const href = linked.links.find(link => hasLink({ ...linked, links: [link] }, "projects", p));
|
||||
const linked = await pollUntil({ label: "Owner handoff project access link rendered", deadlineAt: Date.now() + 20_000,
|
||||
load: () => observeInternal(ctx, state), accept: o => o.links.some(link => isProjectAccessLink(o, link, p)) }).catch(() => first);
|
||||
const accessHref = linked.links.find(link => isProjectAccessLink(linked, link, p)) ?? null;
|
||||
const href = accessHref ?? linked.links.find(link => hasLink({ ...linked, links: [link] }, "projects", p));
|
||||
if (!href) throw new Error("Owner handoff omitted the actual project link");
|
||||
(state.beforeShare as Record<string, unknown>).members = await ctx.input.api.get<AccessMember[]>(`/api/projects/${p.id}/access-members`);
|
||||
(state.beforeShare as Record<string, unknown>).links = linked.links;
|
||||
const sourceUrl = ctx.input.page.url();
|
||||
await internalResultLink(ctx, linked, href).first().click();
|
||||
await ctx.input.page.getByText("Configuration", { exact: true }).last().click();
|
||||
await ctx.input.page.getByRole("button", { name: "Manage access", exact: true }).click();
|
||||
const manageAccess = ctx.input.page.getByRole("button", { name: "Manage access", exact: true });
|
||||
if (accessHref) {
|
||||
await ctx.input.page.waitForURL(url => isProjectAccessLink(linked, url.href, p));
|
||||
await expect(manageAccess).toBeVisible();
|
||||
} else {
|
||||
// Without an access-settings link the grade fails; still complete the owner step so later phases keep their evidence.
|
||||
await ctx.input.page.getByText("Configuration", { exact: true }).last().click();
|
||||
}
|
||||
state.ownerAccessLink = { href: accessHref, opened: !!accessHref };
|
||||
await ctx.input.evidence("chat-catalog-owner-project-access-link.json", { ...(state.ownerAccessLink as object), url: ctx.input.page.url() });
|
||||
await manageAccess.click();
|
||||
const dialog = ctx.input.page.getByRole("dialog", { name: "Private project access", exact: true });
|
||||
// Opening the dialog focuses the person picker, which opens its search panel by itself.
|
||||
// Click the picker only while that panel is still closed; a second click would close it.
|
||||
|
||||
@@ -207,7 +207,9 @@ function valid(scenario: string): InternalObservation {
|
||||
o.state.ownerId = "owner"; o.state.observerId = "riley"; o.state.protectedTask = { id: "protected", visibility: "private" };
|
||||
o.tasks.push({ id: "protected", ...{ visibility: "private" } });
|
||||
const p = { ...o.projects[0]!, name: "Acquisition research", ...{ visibility: "private" as const } }; o.projects[0] = p;
|
||||
o.state.beforeShare = { project: p, observerProjectStatus: 404, protectedTaskStatus: 404, agentMutationStatus: 403, leadReadStatus: 200, reply: "You can use Manage access to share this project with Riley." };
|
||||
o.state.beforeShare = { project: p, observerProjectStatus: 404, protectedTaskStatus: 404, agentMutationStatus: 403, leadReadStatus: 200, reply: "You can use Manage access to share this project with Riley.",
|
||||
members: [{ id: "owner-member", subjectType: "user", subjectId: "owner", source: "owner" }], links: ["/RUN/projects/p/configuration"] };
|
||||
o.state.ownerAccessLink = { href: "/RUN/projects/p/configuration", opened: true };
|
||||
Object.assign(o, { sharedProject: p, members: [{ id: "member", subjectType: "user", subjectId: "riley" }], observerProjectStatus: 200, observerProtectedTaskStatus: 404 });
|
||||
o.activity = [{ ...event("project.access_member_added", "p", { subjectId: "riley" }), agentId: null, actorType: "user", actorId: "owner" }]; break;
|
||||
}
|
||||
@@ -612,9 +614,9 @@ it("rejects a correct clock paired with a wrong named calendar date", () => {
|
||||
const o = valid("activity.audit-change"); o.reply = "Board changed priority to high and low on October 25 at 12:00 UTC.";
|
||||
expect(gradeInternal(o).passed).toBe(false);
|
||||
});
|
||||
it("rejects a premature sharing claim before the owner action", () => {
|
||||
it("rejects a premature grant before the owner action, by saved membership rather than reply wording", () => {
|
||||
const o = valid("projects.share-project");
|
||||
(o.state.beforeShare as { reply: string }).reply = "I've shared it with Riley; you can use Manage access.";
|
||||
(o.state.beforeShare as { members: Entity[] }).members.push({ id: "early", subjectType: "user", subjectId: "riley" } as Entity);
|
||||
expect(gradeInternal(o).passed).toBe(false);
|
||||
});
|
||||
it("accepts the recorded editor named in a sentence form the old phrase list missed", () => {
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { documentContactRemoved, gradeDiscard, promptCarries, replyLinkHrefs } from "./runtime-discard.js";
|
||||
import { contactDetails, contactSectionLines, documentContactRemoved, gradeDiscard, promptCarries, replyLinkHrefs } from "./runtime-discard.js";
|
||||
function valid() {
|
||||
const neighbour = { position: 0, comment: { id: "font", body: "Use a readable font.", authorUserId: "human", authorAgentId: null } };
|
||||
return { scenarioId: "steering.discard-queued-message", origin: "http://fixture.invalid", companyId: "company", leadId: "ceo", agents: [], links: [], humanQueries: [], reply: "",
|
||||
@@ -72,30 +72,62 @@ it("rejects an unbound later worker in the grade when its prompt lacks the direc
|
||||
const o = valid(); o.state.cancelDirectionRunBound = promptCarries(`{"body":${JSON.stringify("unrelated")}}`, "Remove the contact section.\nKeep fonts.");
|
||||
expect(gradeDiscard(o).passed).toBe(false);
|
||||
});
|
||||
const label = "e972d9c1aa95-1";
|
||||
const initialPlan = "# Website Plan (Initial)\n\n## Accessible Font Guidance\n\n- Readable base size.\n\n## Proposed Pages\n1. Home\n2. Contact\n";
|
||||
// Full cohort 11: the consuming run saved the label as a plain line under a "## Contact" heading.
|
||||
const consumedPlan = `# Website Plan (Initial)\n\n## Accessible Font Guidance\n\n- Readable base size.\n\n## Contact\nSection label: \`${label}\`\n\n- Add a contact form.\n\n## Proposed Pages\n1. Home\n2. Contact\n`;
|
||||
const finalPlan = "# Website Plan (Initial)\n\n## Accessible Font Guidance\n\n- Readable base size.\n\n## Proposed Pages\n1. Home\n";
|
||||
const revs = (finalBody: string) => [
|
||||
{ id: "r1", revisionNumber: 1, createdByRunId: "initial", body: initialPlan },
|
||||
{ id: "r2", revisionNumber: 2, createdByRunId: "consumer", body: consumedPlan },
|
||||
{ id: "final", revisionNumber: 3, createdByRunId: "final", body: finalBody },
|
||||
];
|
||||
it("full cohort 11: passes when the final revision drops the setup label, whatever form the label was saved in", () => {
|
||||
expect(documentContactRemoved(revs(finalPlan), "final", label)).toBe(true);
|
||||
for (const form of [`## Contact section (label: ${label})`, `### Reach us (${label})`, `Contact - ${label}`]) {
|
||||
const consumed = consumedPlan.replace(`## Contact\nSection label: \`${label}\``, form);
|
||||
expect(documentContactRemoved([...revs(finalPlan)].map(row => row.id === "r2" ? { ...row, body: consumed } : row), "final", label), form).toBe(true);
|
||||
}
|
||||
const contact = contactDetails("108262186bce-1"), label = "108262186bce-1", details = [contact.name, contact.phone], markers = [label, ...details];
|
||||
const heading = "## Contact Section — `108262186bce-1`";
|
||||
const consumedPlan = [
|
||||
"# Initial Website Plan", "", "Latest direction: add a contact section labelled `108262186bce-1`.", "",
|
||||
"## Proposed Structure", "", "1. **Home** — purpose statement.", "4. **Contact — `108262186bce-1`** — contact form.", "", "---", "",
|
||||
heading, "", `- Contact: ${contact.name}`, `- Phone: ${contact.phone}`, "- Form fields: Name, Email, Message.", "- Use visible labels.", "", "---", "",
|
||||
"## Accessibility Baseline", "", "- Use visible labels.", "",
|
||||
].join("\n");
|
||||
const finalPlan = [
|
||||
"# Initial Website Plan", "", "Latest direction: remove the contact section (label `108262186bce-1`) from the plan.", "",
|
||||
"## Proposed Structure", "", "1. **Home** — purpose statement.", "", "---", "",
|
||||
"## Accessibility Baseline", "", "- Use visible labels.", "",
|
||||
].join("\n");
|
||||
it("collects only the contact section's distinctive whole lines", () => {
|
||||
expect(contactSectionLines(consumedPlan, markers)).toEqual(["Latest direction: add a contact section labelled `108262186bce-1`.",
|
||||
"4. **Contact — `108262186bce-1`** — contact form.", heading, `- Contact: ${contact.name}`, `- Phone: ${contact.phone}`, "- Form fields: Name, Email, Message."]);
|
||||
});
|
||||
it("fails while the final revision still carries the setup label in any form", () => {
|
||||
expect(documentContactRemoved(revs(consumedPlan), "final", label)).toBe(false);
|
||||
expect(documentContactRemoved(revs(`${finalPlan}\nContact (${label}): email us.\n`), "final", label)).toBe(false);
|
||||
it("passes when shared lines remain and a note quotes the label inside a longer line", () => {
|
||||
expect(finalPlan.includes("108262186bce-1")).toBe(true);
|
||||
expect(documentContactRemoved(consumedPlan, finalPlan, label, details)).toBe(true);
|
||||
});
|
||||
it("fails when the final revision is missing or the label is empty", () => {
|
||||
expect(documentContactRemoved(revs(finalPlan), "missing", label)).toBe(false);
|
||||
expect(documentContactRemoved(revs(finalPlan), "final", "")).toBe(false);
|
||||
it("fails when only the contact name and phone are removed", () => {
|
||||
const detailsOnly = consumedPlan.split("\n").filter(line => !line.includes(contact.name) && !line.includes(contact.phone)).join("\n");
|
||||
expect(documentContactRemoved(consumedPlan, detailsOnly, label, details)).toBe(false);
|
||||
});
|
||||
it("fails when the final revision keeps the heading line or a contact detail line", () => {
|
||||
expect(documentContactRemoved(consumedPlan, `${finalPlan}\n${heading}\n`, label, details)).toBe(false);
|
||||
expect(documentContactRemoved(consumedPlan, `${finalPlan}\n - Contact: ${contact.name} \n`, label, details)).toBe(false);
|
||||
});
|
||||
it("fails when the contact name or phone survives in reformatted lines", () => {
|
||||
expect(documentContactRemoved(consumedPlan, `${finalPlan}\n## Contact (${label})\nContact: ${contact.name}\nPhone: ${contact.phone}\n`, label, details)).toBe(false);
|
||||
expect(documentContactRemoved(consumedPlan, `${finalPlan}\nAsk ${contact.name} for help.\n`, label, details)).toBe(false);
|
||||
});
|
||||
it("fails when the final revision is missing or the saved revision has no contact section", () => {
|
||||
expect(documentContactRemoved(consumedPlan, undefined, label, details)).toBe(false);
|
||||
expect(documentContactRemoved(finalPlan.replace(/`108262186bce-1`/g, ""), finalPlan, label, details)).toBe(false);
|
||||
});
|
||||
// Full cohort 14, abridged from the retained revisions (that setup had only the label as a marker).
|
||||
const consumed14 = [
|
||||
"# Initial Website Plan", "", "Latest direction: add a contact section labelled `108262186bce-1`, keep the accessible-font requirement.", "",
|
||||
"## Proposed Structure", "", "1. **Home** — purpose statement.", "4. **Contact — `108262186bce-1`** — accessible contact form with visible labels.", "5. **Legal / Policy** — privacy.", "",
|
||||
"## Contact Section — `108262186bce-1`", "", "- **Purpose:** give visitors a clear, low-friction way to get in touch.",
|
||||
"- **Form fields:** Name, Email, Message — each with a persistent visible `<label>`.", "",
|
||||
"## Accessible Font Guidance (retained)", "", "- Minimum body size of **16px** (`1rem`).", "",
|
||||
"## Open Questions", "", "- Which contact methods and routing should the `108262186bce-1` section use?", "",
|
||||
].join("\n");
|
||||
const final14 = [
|
||||
"# Initial Website Plan", "", "Latest direction: remove the contact section (label `108262186bce-1`) from the plan; keep the rest of the plan unchanged.", "",
|
||||
"## Proposed Structure", "", "1. **Home** — purpose statement.", "4. **Legal / Policy** — privacy.", "",
|
||||
"## Accessible Font Guidance (retained)", "", "- Minimum body size of **16px** (`1rem`).", "",
|
||||
"## Open Questions", "", "- Which pages and content are required at launch?", "",
|
||||
].join("\n");
|
||||
it("passes the retained cohort-14 plan", () => expect(documentContactRemoved(consumed14, final14, label, details)).toBe(true));
|
||||
it("derives the contact details, with an attempt-specific phone", () => {
|
||||
expect(contact).toEqual({ name: "Harriet Quillfeather", phone: "+1-555-1861" });
|
||||
expect(contactDetails("e972d9c1aa95-1").phone).not.toBe(contact.phone);
|
||||
});
|
||||
|
||||
function citedInTurn() {
|
||||
|
||||
@@ -23,12 +23,37 @@ export function replyLinkHrefs(reply: string, rendered: readonly string[]) {
|
||||
return [...new Set([...rendered, ...[...reply.matchAll(/\]\(([^)\s]+)\)/g)].map(match => match[1]!)])];
|
||||
}
|
||||
/**
|
||||
* The cancelled request's setup label must be absent from the document's final revision. Whether the label
|
||||
* was ever saved is checked separately on the consuming run's revisions; the plan's layout is not graded.
|
||||
* Contact details the later request asks for. Only these setup literals identify the contact section. The
|
||||
* composer turns an email address into a mailto link, so the request names a contact person instead.
|
||||
*/
|
||||
export function documentContactRemoved(rows: ReadonlyArray<{ id: string; body?: string | null }>, latestRevisionId: unknown, label: string) {
|
||||
const final = rows.find(row => row.id === latestRevisionId);
|
||||
return !!final && label.length > 0 && !(final.body ?? "").includes(label);
|
||||
export function contactDetails(nonce: string) {
|
||||
return { name: "Harriet Quillfeather", phone: `+1-555-${nonce.replace(/\D/g, "").slice(-4).padStart(4, "0")}` };
|
||||
}
|
||||
/**
|
||||
* Whole trimmed lines that belong only to the contact section of the revision that saved it: the body of a
|
||||
* Markdown section whose heading carries a setup marker (label, contact name or phone), plus lines that carry a
|
||||
* marker. Lines that also appear elsewhere in that revision, such as `---` or a shared bullet, are excluded.
|
||||
*/
|
||||
export function contactSectionLines(body: string, markers: readonly string[]): string[] {
|
||||
const lines = body.split("\n").map(row => row.trim()), marked = (line: string) => markers.some(m => m.length > 0 && line.includes(m));
|
||||
const inSection: string[] = []; let level = 0;
|
||||
for (const line of lines) {
|
||||
const heading = /^(#{1,6})\s/.exec(line)?.[1]?.length;
|
||||
if (heading) level = marked(line) ? heading : heading <= level ? 0 : level;
|
||||
if (line && (level > 0 || marked(line))) inSection.push(line);
|
||||
}
|
||||
const count = (rows: string[], line: string) => rows.filter(row => row === line).length;
|
||||
return [...new Set(inSection)].filter(line => count(lines, line) === count(inSection, line));
|
||||
}
|
||||
/**
|
||||
* None of the contact section's distinctive lines survives as a whole line in the final revision, and the
|
||||
* requested contact name and phone appear nowhere in it, however reformatted. The label alone may still be quoted.
|
||||
*/
|
||||
export function documentContactRemoved(consumedBody: string, finalBody: string | undefined, label: string, details: readonly string[]) {
|
||||
const section = contactSectionLines(consumedBody, [label, ...details]);
|
||||
const kept = new Set((finalBody ?? "").split("\n").map(row => row.trim()));
|
||||
return finalBody !== undefined && section.length > 0 && !section.some(line => kept.has(line))
|
||||
&& !details.some(detail => detail.length > 0 && finalBody.includes(detail));
|
||||
}
|
||||
export function gradeDiscard(value: unknown) {
|
||||
const c = checks(), o = runtimeObservation(value);
|
||||
@@ -90,7 +115,8 @@ export const runtimeDiscardCases = [runtimeDefinition("steering.discard-queued-m
|
||||
await ctx.input.capture("actual-human-contact-discard", "Only the eligible contact request has left the real task queue", "runtime-discard.png");
|
||||
// This new, explicit human request is a distinct later instruction. It
|
||||
// deliberately reaches delivery before its cancellation.
|
||||
const secondText = `I have changed my mind: add a contact section to the saved website plan, labelled ${ctx.input.nonce}. Keep the accessible-font requirement and finish this task.`;
|
||||
const contact = contactDetails(ctx.input.nonce);
|
||||
const secondText = `I have changed my mind: add a contact section to the saved website plan, labelled ${ctx.input.nonce}, listing the contact person ${contact.name} and the phone ${contact.phone}. Keep the accessible-font requirement and finish this task.`;
|
||||
await submitTaskReply(ctx.input.page, secondText);
|
||||
const secondQueue = await pollUntil({ label: "Distinct human contact request genuinely queued", deadlineAt: Date.now() + 20_000,
|
||||
load: () => ctx.input.api.get<IssueQueuedCommentQueue>(queuePath), accept: q => q.entries.length === 2 && q.entries.some(row => row.comment.body === secondText) });
|
||||
@@ -142,14 +168,17 @@ export const runtimeDiscardCases = [runtimeDefinition("steering.discard-queued-m
|
||||
async observe(ctx, state) {
|
||||
const o = await observeRuntime(ctx, state);
|
||||
state.initialRevisionRetained = o.revisions.some(row => row.createdByRunId === state.workerRunId && row.createdByAgentId === (state.worker as Entity).id);
|
||||
// The consumed request asked for a contact section labelled with the attempt nonce. Its
|
||||
// consuming run must have saved that label, and the cancellation run's latest plan must drop it.
|
||||
const label = ctx.input.nonce, worker = (state.worker as Entity).id;
|
||||
state.consumedLabelSaved = o.revisions.some(row => row.createdByRunId === state.consumerRunId && row.createdByAgentId === worker && (row.body ?? "").includes(label));
|
||||
// The consumed request asked for a contact section with the setup's contact name and phone. Its consuming run
|
||||
// must have saved the name, and the cancellation run's latest plan must drop that section's distinctive lines.
|
||||
const contact = contactDetails(ctx.input.nonce), worker = (state.worker as Entity).id;
|
||||
const consumed = o.revisions.filter(row => row.createdByRunId === state.consumerRunId && row.createdByAgentId === worker && (row.body ?? "").includes(contact.name));
|
||||
state.consumedLabelSaved = consumed.length > 0;
|
||||
const latest = o.revisions.filter(row => row.createdByRunId === state.finalWorkerRunId && row.createdByAgentId === worker
|
||||
&& o.documents.some(doc => doc.id === row.documentId && doc.latestRevisionId === row.id));
|
||||
state.finalPlanWithoutContact = latest.length > 0 && o.documents.every(doc => documentContactRemoved(
|
||||
o.revisions.filter(row => row.documentId === doc.id), doc.latestRevisionId, label));
|
||||
state.finalPlanWithoutContact = latest.length > 0 && consumed.every(row => {
|
||||
const doc = o.documents.find(d => d.id === row.documentId);
|
||||
return documentContactRemoved(row.body ?? "", o.revisions.find(r => r.id === doc?.latestRevisionId)?.body ?? undefined, ctx.input.nonce, [contact.name, contact.phone]);
|
||||
});
|
||||
return o;
|
||||
},
|
||||
async cleanup(ctx, state) { try { await cleanupIdentityProbes(ctx); } finally { await releaseRuntimeWorker(state); } },
|
||||
|
||||
@@ -25,3 +25,10 @@ it("does not require the word uncommitted when HEAD and status prove it, but sti
|
||||
expect(gradeInspectGit(o).passed).toBe(true);
|
||||
o.reply += "\nI've committed these changes."; expect(gradeInspectGit(o).passed).toBe(false);
|
||||
});
|
||||
it("accepts the retained cohort-14 reply, which leaves out the unrelated non-website file", () => {
|
||||
const o = valid();
|
||||
o.reply = "Here is what changed in the website files:\n\n**header.html** (modified)\n```diff\n-<h1>Launch later</h1>\n+<h1>Launch sooner</h1>\n```\n\n**website-fixture.txt** (new file)\n```\nSaved website prerequisite 4c9950ba26fb-1\n```\n\nThe header heading changed from \"Launch later\" to \"Launch sooner\", and a new fixture file was added. No other website files were touched.";
|
||||
expect(gradeInspectGit(o).passed).toBe(true);
|
||||
// Leaving the unrelated file out of the answer never excuses changing it.
|
||||
o.state.notesAfter = "Rewritten notes"; expect(gradeInspectGit(o).passed).toBe(false);
|
||||
});
|
||||
@@ -29,10 +29,10 @@ export function gradeInspectGit(value: unknown) {
|
||||
c.add("actual-diff-summary-with-file-references", /header\.html/.test(o.reply) && /Launch later/i.test(o.reply) && /Launch sooner/i.test(o.reply)
|
||||
// The changed files and values are the setup's own literals, so the reply must report them. That the work
|
||||
// is uncommitted is proven by the unchanged HEAD/status above; the reply need not use the word, only
|
||||
// make no commit/push claim.
|
||||
&& /existing-notes\.txt/.test(o.reply)
|
||||
// make no commit/push claim. The person asked about the website files, so the reply may leave out the
|
||||
// unrelated existing-notes.txt change; its preservation is graded byte-for-byte above.
|
||||
&& !/(?:I(?:'ve| have)? |successfully )(?:pushed|committed|published|deployed)|remote.*(?:has been|was) updated/i.test(o.reply),
|
||||
"The answer describes the observed header and unrelated diff and makes no commit, push or remote-update claim.");
|
||||
"The answer describes the observed header diff and makes no commit, push or remote-update claim.");
|
||||
c.add("real-native-inspection-only", o.runs.length === 2 && o.runs.every(run => run.status === "succeeded")
|
||||
&& o.runs.filter(run => run.agentId === o.leadId).length === 1,
|
||||
"The real prior workspace setup and one native inspection are the only provider executions.");
|
||||
|
||||
@@ -25,7 +25,7 @@ function fixture(options: { loadingForever?: boolean; extraRun?: boolean; extraC
|
||||
return kind === "heading" || ready && !(kind.includes("comment-reply") && options.missingRenderedReply);
|
||||
}, editable: () => ready && !options.readOnly });
|
||||
const page = { url: () => current, goto: async (url: string) => { current = new URL(url, "https://fixture.test").href; ready = false; },
|
||||
reload: async () => {}, getByRole: () => locator("heading"), getByTestId: () => locator("composer"), locator };
|
||||
reload: async () => {}, waitForURL: async (matches: (url: URL) => boolean) => { if (!matches(new URL(current))) throw Error("Profile route never settled"); }, getByRole: () => locator("heading"), getByTestId: () => locator("composer"), locator };
|
||||
const api = { baseURL: "https://fixture.test", get: async (path: string) => {
|
||||
calls.push(path); if (path.endsWith("/primary-agent/me")) return primary;
|
||||
if (path.endsWith("/chats/ceo")) return { ...source, ...(options.wrongChat ? { id: "different-source" } : {}) };
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { chromium } from "@playwright/test";
|
||||
import { expect, it } from "vitest";
|
||||
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
||||
import { openRuntimeAgentProfile, type RuntimeAgent } from "./runtime-support.js";
|
||||
|
||||
const executablePath = process.env.PAPERCLIP_RUNNER_E2E_CHROMIUM_EXECUTABLE ?? chromium.executablePath();
|
||||
const origin = "https://paperclip.example";
|
||||
const agent: RuntimeAgent = { id: "8f2b6c1e-0000-4000-8000-000000000001", name: "Morgan", urlKey: "morgan" };
|
||||
|
||||
// The profile is first served on the id route; after a delay the app replaces it with the urlKey route and
|
||||
// redraws, closing any open menu (cohort 14, runs-adapters.reset-session). The menu item renders 300 ms
|
||||
// after the menu opens, so a menu opened before the redirect never shows "Reset sessions".
|
||||
const profile = (redirectAfterMs: number) => `<header><img alt="Morgan avatar" src="data:,"></header><main id="main"></main><script>
|
||||
const draw = () => { document.querySelector("#main").innerHTML = '<button aria-label="Open actions for Morgan" id="open">⋯</button><div id="menu"></div>';
|
||||
document.querySelector("#open").onclick = () => { const menu = document.querySelector("#menu"); setTimeout(() => { const item = document.createElement("button"); item.textContent = "Reset sessions";
|
||||
item.onclick = () => { document.title = "reset"; }; menu.append(item); }, 300); }; };
|
||||
draw();
|
||||
setTimeout(() => { history.replaceState(null, "", "/COM/agents/morgan/dashboard"); draw(); }, ${redirectAfterMs});
|
||||
</script>`;
|
||||
|
||||
it.skipIf(!existsSync(executablePath))("opens the agent's actions menu only after the profile settles on its canonical route", async () => {
|
||||
const browser = await chromium.launch({ headless: true, executablePath });
|
||||
try {
|
||||
for (const redirectAfterMs of [0, 400, 1_500]) {
|
||||
const page = await browser.newPage({ baseURL: origin });
|
||||
await page.route("**/*", route => route.fulfill({ contentType: "text/html; charset=utf-8", body: profile(redirectAfterMs) }));
|
||||
const ctx = { input: { page, fixtures: { company: { issuePrefix: "COM" } }, api: { get: async () => { throw new Error("Unexpected API read"); } } } } as unknown as ChatCatalogContext;
|
||||
await openRuntimeAgentProfile(ctx, agent);
|
||||
await page.getByRole("main").getByRole("button", { name: "Open actions for Morgan", exact: true }).click();
|
||||
await page.getByRole("button", { name: "Reset sessions" }).click({ timeout: 3_000 });
|
||||
expect(await page.title(), String(redirectAfterMs)).toBe("reset");
|
||||
expect(new URL(page.url()).pathname, String(redirectAfterMs)).toBe("/COM/agents/morgan/dashboard");
|
||||
await page.close();
|
||||
}
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
}, 30_000);
|
||||
@@ -1,4 +1,5 @@
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import { deriveAgentUrlKey } from "../../../packages/shared/src/agent-url-key.js";
|
||||
import { pollUntil } from "../api.js";
|
||||
import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contract.js";
|
||||
import { codexSubscriptionAgentConfig } from "../codex-subscription-fixture.js";
|
||||
@@ -59,8 +60,20 @@ export async function openRuntimeAgentProfile(ctx: ChatCatalogContext, agent: Ru
|
||||
const href = o && agentLink(o, agent);
|
||||
if (href) await ctx.input.page.locator(`[id^="comment-"] a[href=${JSON.stringify(href)}]`).first().click();
|
||||
else await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/agents/${agent.id}`, { waitUntil: "domcontentloaded" });
|
||||
// The same route key the app's agentRouteRef chooses.
|
||||
await settleAgentProfileRoute(ctx.input.page, agent.urlKey ?? deriveAgentUrlKey(agent.name, agent.id));
|
||||
await expect(agentProfileIdentity(ctx.input.page, agent)).toBeVisible();
|
||||
}
|
||||
/**
|
||||
* The profile page rewrites `/agents/<id>/…` to `/agents/<urlKey>/…` with a client-side replace, which
|
||||
* redraws the page and closes any open menu. Wait for that canonical route before using its controls.
|
||||
*/
|
||||
export async function settleAgentProfileRoute(page: Page, ref: string, options: { timeout?: number } = {}) {
|
||||
await page.waitForURL(url => {
|
||||
const parts = url.pathname.split("/").map(decodeURIComponent); const at = parts.indexOf("agents");
|
||||
return at >= 0 && parts[at + 1] === ref;
|
||||
}, options);
|
||||
}
|
||||
/**
|
||||
* The profile header's h1 is the agent's editable title (falling back to its
|
||||
* role), and the name is secondary text. The header portrait carries the
|
||||
|
||||
Reference in new issue
Block a user