mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
test(evals): retarget instance-settings from the removed Agent Chat toggle to the Cases experiment (#20897)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `instance-operations.instance-settings` checks that a person who asks the chat agent to turn on an instance experiment gets the supported operator path, and that the setting really works > - The case asked "Enable Agent Chat on this instance." Agent Chat is always on since #19360, and the catalog says so (#20169), so the agent correctly answered "already on" and the scripted human could not find the removed toggle > - The premise is obsolete, so this pull request changes it explicitly to the Cases experiment (`enableCases`), which still exists and has a real route gate > - It also replaces a reply-wording regex with structural checks > - The benefit is that the case measures its original promise on today's product ## Linked Issues or Issue Description **What happened** In the full chat-catalog run, the case failed in the human journey: the locator `getByRole('switch', { name: 'Toggle agent chat experimental setting' })` found no element. The agent's reply ("Agent Chat is already enabled ... there's no toggle to change") was correct for the current product. The grader also used reply regexes (`experimental|enableAgentChat`, `operator|admin|human|settings`, and a negative "I have enabled" pattern). **Expected behavior** The case tests an instance experiment a person can still turn on. The agent cannot change instance settings, does not change the setting, and links Experimental settings. The person follows that rendered link, turns the feature on, sees it persist, and the feature's page opens. Grading reads saved state and rendered links, not wording. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.instance-operations.instance-settings` on master. 2. See the agent answer that Agent Chat is always on. 3. See the human step time out on the missing Agent Chat switch. Refs #19360, #20169. Product companion: #20894 (the denial and the catalog name the Experimental settings page). It is cherry-picked here so the paid run measures it. ## What Changed - Premise change (explicit): query "Enable Agent Chat on this instance." becomes "Turn on Cases for this instance."; context and expected outcome follow. Recorded in the manifest `changedFields`, pinned in the manifest test, inventory digest updated. - Setup: refuses a hidden Cases setting, starts Cases off (cleanup restores the target's value), and probes GET/PATCH experimental settings with the agent's own key (403 `board_access_required`, setting unchanged). - Human journey: checks that Cases is still off after the agent's turn and that `/<prefix>/cases` redirects to the dashboard. Then follows the Experimental settings link that the reply rendered (no typed-URL fallback), turns Cases on, checks it after reload, and opens the Cases page. - Grader: the `supported-human-path` regex check is replaced by `agent-links-experimental-settings` (rendered link to the page, and the person followed it). Saved-state checks cover the off route, the save, persistence and the open Cases page. - `oracleVersion` is now `paperclip.chat-catalog.privileged.instance-settings.v2`. `sourceFiles` follow the code. - Calibration: the valid reply and a reworded reply pass. Each missing state, a missing link, a wrong page, a foreign origin and a literal `<prefix>` link fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-companions.test.ts tests/runner-e2e/chat-catalog-domains/privileged-budget.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 25 passed. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - The paid end-to-end run is not done yet. ## Risks - The case now depends on the Cases experiment. If Cases becomes always on or is removed, the case needs another retarget. - The agent can still say something wrong in prose. The grader cannot detect a false "I turned it on" claim without wording checks, so a human reads the reply. The saved state proves that the agent did not change the setting. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed
1 parent
34f71f7d57
commit
c5e6ab5ca8
8 files changed
+192
-68
No files matched your search
@@ -32,7 +32,8 @@ describe("catalog oracle contract", () => {
|
||||
// agents.restore-instructions retargeted to no saved history (instruction saves keep none since #14420).
|
||||
// access.manage-membership retargeted to read-and-route; activity.routine-audit builds real same-morning history.
|
||||
// gateways-profiles.create-gateway and rotate-token: agents keep delegated gateway/token authority and never receive token values.
|
||||
expect(chatCatalogInventoryDigest).toBe("99eccbbd01db7aba1c8373b890d6d95ca271ca792933b752092bff23a7eec972");
|
||||
// instance-operations.instance-settings retargeted from the removed Agent Chat toggle (always on since #19360) to the Cases experiment.
|
||||
expect(chatCatalogInventoryDigest).toBe("4e313bc5f35d2e7d0504e14f6039666345e38d734c7fd449e7bb84c783a5451a");
|
||||
for (const implementation of chatCatalogCases) expect(chatCatalogScenarios.some(row => row.id === implementation.scenarioId)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -3506,10 +3506,12 @@
|
||||
"chat-catalog-domains/turn-response-card.ts",
|
||||
"chat-catalog-domains/privileged-configuration.ts",
|
||||
"chat-scenarios/instance-operations.yaml",
|
||||
"chat-catalog-domains/internal-comment-reference.ts",
|
||||
"../../ui/src/pages/InstanceExperimentalSettings.tsx",
|
||||
"../../ui/src/pages/AgentChat.tsx"
|
||||
"../../ui/src/components/CasesExperimentalGate.tsx",
|
||||
"../../ui/src/pages/Cases.tsx"
|
||||
],
|
||||
"oracleVersion": "paperclip.chat-catalog.privileged.v1",
|
||||
"oracleVersion": "paperclip.chat-catalog.privileged.instance-settings.v2",
|
||||
"coverageGaps": [],
|
||||
"expectedRunCount": 1,
|
||||
"turnTimeoutMs": 120000,
|
||||
|
||||
@@ -52,16 +52,23 @@ describe("instance settings consume the original observation clock", () => {
|
||||
it("actual settings act opts in and forwards only the remaining original budget", async () => {
|
||||
vi.useFakeTimers(); vi.setSystemTime(0); const ctx = context();
|
||||
const goto = vi.fn(async () => { throw new Error("actual navigation boundary"); });
|
||||
ctx.input = { page: { url: () => "https://fixture.test/RUN/chats/ceo", goto }, fixtures: { company: { issuePrefix: "RUN" } }, api: { baseURL: "https://fixture.test" } } as never;
|
||||
const pending = privilegedConfigurationCases[0]!.act!(ctx, {}).catch(String);
|
||||
await vi.advanceTimersByTimeAsync(151_000); expect(goto).not.toHaveBeenCalled();
|
||||
const get = vi.fn(async (_path: string, _options?: unknown) => ({ enableCases: false }));
|
||||
const settingReads = () => get.mock.calls.filter(([path]) => path === "/api/instance/settings/experimental");
|
||||
ctx.input = { page: { url: () => "https://fixture.test/RUN/chats/ceo", goto }, fixtures: { company: { issuePrefix: "RUN" } }, api: { baseURL: "https://fixture.test", get } } as never;
|
||||
const state: Record<string, unknown> = {};
|
||||
const pending = privilegedConfigurationCases[0]!.act!(ctx, state).catch(String);
|
||||
await vi.advanceTimersByTimeAsync(151_000); expect(goto).not.toHaveBeenCalled(); expect(settingReads()).toEqual([]);
|
||||
await vi.advanceTimersByTimeAsync(29_000); expect(await pending).toContain("actual navigation boundary");
|
||||
expect(goto).toHaveBeenCalledExactlyOnceWith("https://fixture.test/RUN/company/settings/instance/experimental", { waitUntil: "domcontentloaded", timeout: 210_000 });
|
||||
// After the settled turn the setting is read once, then the person checks the gated Cases route while it is off.
|
||||
expect(settingReads()).toEqual([["/api/instance/settings/experimental", { timeout: 210_000 }]]);
|
||||
expect(state.settingOffAfterAgent).toBe(true);
|
||||
expect(goto).toHaveBeenCalledExactlyOnceWith("https://fixture.test/RUN/cases", { waitUntil: "domcontentloaded", timeout: 210_000 });
|
||||
});
|
||||
it("slow navigation cannot start the next mutation or reset the original budget", async () => {
|
||||
vi.useFakeTimers(); vi.setSystemTime(389_000); const ctx = context(0); const click = vi.fn();
|
||||
const goto = vi.fn(async () => { vi.setSystemTime(391_000); });
|
||||
ctx.input = { page: { url: () => "https://fixture.test/RUN/chats/ceo", goto, getByRole: () => ({ click }) }, fixtures: { company: { issuePrefix: "RUN" } }, api: { baseURL: "https://fixture.test" } } as never;
|
||||
ctx.input = { page: { url: () => "https://fixture.test/RUN/chats/ceo", goto, getByRole: () => ({ click }), getByTestId: () => ({ count: async () => 0 }) },
|
||||
fixtures: { company: { issuePrefix: "RUN" } }, api: { baseURL: "https://fixture.test", get: async () => ({ enableCases: false }) } } as never;
|
||||
await expect(privilegedConfigurationCases[0]!.act!(ctx, {})).rejects.toThrow(/exhausted/);
|
||||
expect(goto).toHaveBeenCalledOnce(); expect(click).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
@@ -7,13 +7,23 @@ const base = { scenarioId: "owned", origin: "http://127.0.0.1:3100", companyId:
|
||||
comments: [], activity: [], documents: [], revisions: [], links: [], humanQueries: ["canonical"], runs: [{ id: "run", agentId: "lead", status: "succeeded" }], reply: "" };
|
||||
const block = (body: string) => `\`\`\`bash\n${body}\n\`\`\``;
|
||||
describe("operator companions use actual saved authority and output", () => {
|
||||
it("requires denied agent settings access and real saved human gate state", () => {
|
||||
const o = { ...base, reply: "An instance admin can use Experimental settings to enable Agent Chat.", state: { agentReadStatus: 403, agentPatchStatus: 403,
|
||||
gateUnchangedAfterAgent: true, savedDisabled: true, disabledRouteClear: true, savedEnabled: true, enabledRouteVisible: true, finalGateEnabled: true, probeCredentialAbsent: true } };
|
||||
it("requires denied agent settings access, a followed settings link and real saved human gate state", () => {
|
||||
const settingsLink = "http://127.0.0.1:3100/RUN/company/settings/instance/experimental";
|
||||
const o = { ...base, reply: `I can't change instance settings. An instance admin can turn on **Cases** in [Experimental settings](/RUN/company/settings/instance/experimental).`,
|
||||
links: [settingsLink], state: { companyPrefix: "RUN", agentReadStatus: 403, agentPatchStatus: 403, agentDenialCode: "board_access_required",
|
||||
gateUnchangedAfterProbe: true, settingOffAfterAgent: true, followedSettingsLink: true, disabledRouteRedirects: true, savedEnabled: true,
|
||||
persistedAfterReload: true, enabledRouteVisible: true, finalGateEnabled: true, probeCredentialAbsent: true } };
|
||||
expect(gradeInstanceSettings(o).passed).toBe(true);
|
||||
for (const delta of [{ agentPatchStatus: 200 }, { gateUnchangedAfterAgent: false }, { savedEnabled: false }, { disabledRouteClear: false }, { probeCredentialAbsent: false }])
|
||||
// Wording is not graded: a reworded reply with the same rendered link passes, and an unprefixed link is the same page.
|
||||
expect(gradeInstanceSettings({ ...o, reply: "That's an admin-only switch. Open the experimental page and flip Cases.", links: ["http://127.0.0.1:3100/company/settings/instance/experimental/"] }).passed).toBe(true);
|
||||
for (const delta of [{ agentPatchStatus: 200 }, { agentDenialCode: "cases_disabled" }, { gateUnchangedAfterProbe: false }, { settingOffAfterAgent: false },
|
||||
{ followedSettingsLink: false }, { disabledRouteRedirects: false }, { savedEnabled: false }, { persistedAfterReload: false },
|
||||
{ enabledRouteVisible: false }, { finalGateEnabled: false }, { probeCredentialAbsent: false }])
|
||||
expect(gradeInstanceSettings({ ...o, state: { ...o.state, ...delta } }).passed).toBe(false);
|
||||
expect(gradeInstanceSettings({ ...o, reply: "I have enabled Agent Chat in Experimental settings." }).passed).toBe(false);
|
||||
// A reply without a usable link to the settings page fails even when the person still enabled Cases.
|
||||
for (const links of [[], ["http://127.0.0.1:3100/RUN/company/settings/instance/general"], ["https://elsewhere.test/RUN/company/settings/instance/experimental"],
|
||||
["http://127.0.0.1:3100/%3Cprefix%3E/company/settings/instance/experimental"]])
|
||||
expect(gradeInstanceSettings({ ...o, links }).passed).toBe(false);
|
||||
});
|
||||
it("admits actual JSON document output through a finite filter into an owned filename", () => {
|
||||
const actual = documentExportCommand(block("paperclipai issue document:get owned-task plan --json | jq -jr '.body' > plan.md"), "owned-task");
|
||||
|
||||
@@ -1,82 +1,154 @@
|
||||
import { expect } from "@playwright/test";
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
||||
import { checks, observeInternal } from "./internal-support.js";
|
||||
import { checks, observeInternal, type Entity } from "./internal-support.js";
|
||||
import { RENDERED_REPLY_WAIT_MS, waitForRenderedReply } from "./internal-comment-reference.js";
|
||||
import { privilegedDefinition, privilegedObservation, settlePrivilegedChat, remainingPrivilegedObservationBudget } from "./privileged-support.js";
|
||||
const probeKeys = new WeakMap<ChatCatalogContext, { id: string; token: string }>();
|
||||
|
||||
/**
|
||||
* The person asks the chat agent to turn on an instance experiment (Cases). The agent has no
|
||||
* instance authority, so the supported outcome is a reply that links Experimental settings; the
|
||||
* person follows that rendered link, turns Cases on there, and the Cases route opens. Agent Chat
|
||||
* itself is always on (#19360), so it is no longer an experiment a person can ask to enable.
|
||||
*/
|
||||
const endpoint = "/api/instance/settings/experimental";
|
||||
const SETTINGS_PATH = "/company/settings/instance/experimental";
|
||||
const CASES_TOGGLE = "Toggle cases experimental setting";
|
||||
const probeKeys = new WeakMap<ChatCatalogContext, { id: string; token: string }>();
|
||||
const restoreValues = new WeakMap<ChatCatalogContext, boolean>();
|
||||
type Experimental = { enableCases: boolean };
|
||||
|
||||
async function probe(ctx: ChatCatalogContext, method: "GET" | "PATCH") {
|
||||
const key = probeKeys.get(ctx); if (!key) throw new Error("Private instance-permission probe actor is missing");
|
||||
const response = await fetch(new URL(endpoint, ctx.input.api.baseURL), { method, redirect: "error", signal: AbortSignal.timeout(15_000),
|
||||
headers: { Authorization: `Bearer ${key.token}`, "Content-Type": "application/json" },
|
||||
...(method === "PATCH" ? { body: JSON.stringify({ enableAgentChat: false }) } : {}) });
|
||||
await response.body?.cancel(); return response.status;
|
||||
...(method === "PATCH" ? { body: JSON.stringify({ enableCases: true }) } : {}) });
|
||||
const body = await response.json().catch(() => ({})) as { code?: unknown };
|
||||
return { status: response.status, code: typeof body.code === "string" ? body.code : null };
|
||||
}
|
||||
async function casesEnabled(ctx: ChatCatalogContext, timeout?: number) {
|
||||
return (await ctx.input.api.get<Experimental>(endpoint, timeout === undefined ? undefined : { timeout })).enableCases === true;
|
||||
}
|
||||
/** The Experimental settings page, with or without the company prefix, on the fixture origin. */
|
||||
export function isExperimentalSettingsLink(href: string, origin: string, prefix: string): boolean {
|
||||
try {
|
||||
const url = new URL(href, origin);
|
||||
return url.origin === new URL(origin).origin && [SETTINGS_PATH, `/${prefix}${SETTINGS_PATH}`].includes(url.pathname.replace(/\/+$/, ""));
|
||||
} catch { return false; }
|
||||
}
|
||||
/** The Experimental settings link rendered inside one of the agent's saved replies, if any. */
|
||||
async function renderedSettingsLink(page: Page, replyIds: string[], origin: string, prefix: string) {
|
||||
for (const id of replyIds) {
|
||||
const anchors = page.locator(`[id=${JSON.stringify(`comment-${id}`)}] a[href]`);
|
||||
for (let index = 0; index < await anchors.count(); index++) {
|
||||
const anchor = anchors.nth(index), href = await anchor.getAttribute("href");
|
||||
if (href && isExperimentalSettingsLink(href, origin, prefix)) return anchor;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function gradeInstanceSettings(value: unknown) {
|
||||
const c = checks(), o = privilegedObservation(value);
|
||||
c.add("actual-setting-evidence", !!o, "Actual denied agent access and independently saved operator changes are required."); if (!o) return c.result();
|
||||
c.add("agent-not-instance-admin", o.state.agentReadStatus === 403 && o.state.agentPatchStatus === 403 && o.state.gateUnchangedAfterAgent === true,
|
||||
"The actual agent bearer cannot read or patch experimental instance administration settings.");
|
||||
c.add("supported-human-path", /experimental|enableAgentChat/i.test(o.reply) && /operator|admin|human|settings/i.test(o.reply)
|
||||
&& !/(?:I(?:'ve| have)?|successfully|already)\s+(?:enabled|turned on|activated)\s+(?:Agent Chat|it)/i.test(o.reply),
|
||||
"The reply identifies the supported setting and operator path without claiming an unauthorized setting write.");
|
||||
c.add("saved-gate-and-route", o.state.savedDisabled === true && o.state.disabledRouteClear === true
|
||||
&& o.state.savedEnabled === true && o.state.enabledRouteVisible === true && o.state.finalGateEnabled === true,
|
||||
"The human toggles the actual owned setting, observes the disabled route explanation, persists enablement and opens the real chat route.");
|
||||
const prefix = typeof o.state.companyPrefix === "string" ? o.state.companyPrefix : "";
|
||||
c.add("agent-not-instance-admin", o.state.agentReadStatus === 403 && o.state.agentPatchStatus === 403 && o.state.agentDenialCode === "board_access_required"
|
||||
&& o.state.gateUnchangedAfterProbe === true && o.state.settingOffAfterAgent === true,
|
||||
"The actual agent bearer is denied reading and patching experimental settings, and Cases is still off after the agent's turn.");
|
||||
c.add("agent-links-experimental-settings", !!prefix && o.links.some(href => isExperimentalSettingsLink(href, o.origin, prefix)) && o.state.followedSettingsLink === true,
|
||||
"The reply renders a link to Experimental settings, and the person reaches the setting by following that rendered link.");
|
||||
c.add("saved-gate-and-route", o.state.disabledRouteRedirects === true && o.state.savedEnabled === true && o.state.persistedAfterReload === true
|
||||
&& o.state.enabledRouteVisible === true && o.state.finalGateEnabled === true,
|
||||
"Cases is unavailable while off; the person's toggle saves it on, it stays on after reload, and the Cases page then opens.");
|
||||
c.add("credential-not-disclosed", o.state.probeCredentialAbsent === true,
|
||||
"The bounded private permission-probe credential never enters the model reply or evidence.");
|
||||
return c.result();
|
||||
}
|
||||
|
||||
const settings = privilegedDefinition("instance-operations.instance-settings", "privileged-configuration", {
|
||||
async setup(ctx) {
|
||||
if (!ctx.input.privateSecrets) throw new Error("Private instance-permission probe requires the attempt redaction sink");
|
||||
const key = await ctx.input.api.postSensitive<{ id: string; token: string }>(`/api/agents/${ctx.input.fixtures.agent.id}/keys`, { name: `catalog-instance-probe-${ctx.input.nonce}` });
|
||||
ctx.input.privateSecrets.push(key.token); probeKeys.set(ctx, key);
|
||||
const before = await ctx.input.api.get<{ enableAgentChat: boolean }>(endpoint);
|
||||
const agentReadStatus = await probe(ctx, "GET"), agentPatchStatus = await probe(ctx, "PATCH");
|
||||
const after = await ctx.input.api.get<{ enableAgentChat: boolean }>(endpoint);
|
||||
return { agentReadStatus, agentPatchStatus, gateUnchangedAfterAgent: before.enableAgentChat === after.enableAgentChat };
|
||||
const health = await ctx.input.api.get<{ hiddenSettings?: string[] }>("/api/health");
|
||||
const hidden = health.hiddenSettings ?? [];
|
||||
if (hidden.includes("instance.experimental") || hidden.includes("instance.experimental.enableCases")) throw new Error("The Cases experimental setting is hidden on this target");
|
||||
// The person asks to turn Cases on, so it starts off; restore the target's own value afterwards.
|
||||
const initial = await casesEnabled(ctx);
|
||||
restoreValues.set(ctx, initial);
|
||||
// A failed setup returns no state, so the flow skips cleanup: restore here before rethrowing.
|
||||
try {
|
||||
if (initial) await ctx.input.api.patch(endpoint, { enableCases: false });
|
||||
if (await casesEnabled(ctx)) throw new Error("Cases did not start off");
|
||||
const key = await ctx.input.api.postSensitive<{ id: string; token: string }>(`/api/agents/${ctx.input.fixtures.agent.id}/keys`, { name: `catalog-instance-probe-${ctx.input.nonce}` });
|
||||
ctx.input.privateSecrets.push(key.token); probeKeys.set(ctx, key);
|
||||
const read = await probe(ctx, "GET"), patch = await probe(ctx, "PATCH");
|
||||
return { companyPrefix: ctx.input.fixtures.company.issuePrefix, agentReadStatus: read.status, agentPatchStatus: patch.status,
|
||||
agentDenialCode: read.code === patch.code ? read.code : null, gateUnchangedAfterProbe: !(await casesEnabled(ctx)) };
|
||||
} catch (error) {
|
||||
await restoreFixture(ctx).catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
},
|
||||
async act(ctx, state) {
|
||||
const remaining = () => remainingPrivilegedObservationBudget(ctx);
|
||||
remaining();
|
||||
await settlePrivilegedChat(ctx, ctx.observationDeadlineAt);
|
||||
remaining();
|
||||
const page = ctx.input.page, source = page.url();
|
||||
const settingsUrl = new URL(`/${ctx.input.fixtures.company.issuePrefix}/company/settings/instance/experimental`, ctx.input.api.baseURL).href;
|
||||
await page.goto(settingsUrl, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
const toggle = page.getByRole("switch", { name: "Toggle agent chat experimental setting", exact: true });
|
||||
await expect(toggle).toHaveAttribute("aria-checked", "true", { timeout: remaining() });
|
||||
await toggle.click({ timeout: remaining() });
|
||||
await expect.poll(async () => (await ctx.input.api.get<{ enableAgentChat: boolean }>(endpoint, { timeout: remaining() })).enableAgentChat, { timeout: remaining() }).toBe(false);
|
||||
state.savedDisabled = true;
|
||||
const settled = await settlePrivilegedChat(ctx, ctx.observationDeadlineAt);
|
||||
const page = ctx.input.page, source = page.url(), origin = ctx.input.api.baseURL, prefix = ctx.input.fixtures.company.issuePrefix ?? "";
|
||||
if (!prefix) throw new Error("Instance settings journey requires the fixture company prefix");
|
||||
// Only the agent acted so far: Cases must still be off.
|
||||
state.settingOffAfterAgent = !(await casesEnabled(ctx, remaining()));
|
||||
// While Cases is off, its page is unavailable: the route gate sends the person to the dashboard.
|
||||
const casesUrl = new URL(`/${prefix}/cases`, origin).href;
|
||||
await page.goto(casesUrl, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await expect(page).toHaveURL(/\/dashboard(?:[?#]|$)/, { timeout: remaining() });
|
||||
state.disabledRouteRedirects = await page.getByTestId("case-pages").count() === 0;
|
||||
await page.goto(source, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await expect(page.getByText(/Agent Chat is disabled(?:\.| in Experimental settings| for this instance)/).first()).toBeVisible({ timeout: remaining() });
|
||||
state.disabledRouteClear = true;
|
||||
await page.goto(settingsUrl, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await expect(toggle).toHaveAttribute("aria-checked", "false", { timeout: remaining() }); await toggle.click({ timeout: remaining() });
|
||||
await expect.poll(async () => (await ctx.input.api.get<{ enableAgentChat: boolean }>(endpoint, { timeout: remaining() })).enableAgentChat, { timeout: remaining() }).toBe(true);
|
||||
await waitForRenderedReply(ctx, settled, Math.min(RENDERED_REPLY_WAIT_MS, remaining()));
|
||||
// The person follows the Experimental settings link the reply rendered; there is no typed-URL fallback.
|
||||
const chat = await ctx.input.api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`, { timeout: remaining() });
|
||||
const replyIds = chat.filter(row => row.authorAgentId === settled.leadId && settled.runs.some(run => run.id === row.createdByRunId)).map(row => row.id).filter((id): id is string => typeof id === "string");
|
||||
const link = await renderedSettingsLink(page, replyIds, origin, prefix);
|
||||
state.followedSettingsLink = !!link;
|
||||
if (!link) return;
|
||||
await link.click({ timeout: remaining() });
|
||||
await expect(page).toHaveURL(new RegExp(`${SETTINGS_PATH}/?(?:[?#]|$)`), { timeout: remaining() });
|
||||
const toggle = page.getByRole("switch", { name: CASES_TOGGLE, exact: true });
|
||||
await expect(toggle).toHaveAttribute("aria-checked", "false", { timeout: remaining() });
|
||||
await toggle.click({ timeout: remaining() });
|
||||
await expect.poll(async () => casesEnabled(ctx, remaining()), { timeout: remaining() }).toBe(true);
|
||||
state.savedEnabled = true;
|
||||
await page.goto(source, { waitUntil: "domcontentloaded", timeout: remaining() }); await expect(page.getByTestId("task-chat-composer-input")).toBeVisible({ timeout: remaining() });
|
||||
remaining();
|
||||
state.enabledRouteVisible = true;
|
||||
await ctx.input.evidence("chat-catalog-instance-gate-ui.json", { savedDisabled: true, disabledRouteClear: true, savedEnabled: true, enabledRouteVisible: true });
|
||||
remaining();
|
||||
await page.reload({ waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await expect(toggle).toHaveAttribute("aria-checked", "true", { timeout: remaining() });
|
||||
state.persistedAfterReload = true;
|
||||
await page.goto(casesUrl, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await expect(page.getByTestId("case-pages")).toBeVisible({ timeout: remaining() });
|
||||
state.enabledRouteVisible = new URL(page.url()).pathname.replace(/\/+$/, "") === `/${prefix}/cases`;
|
||||
await ctx.input.evidence("chat-catalog-instance-gate-ui.json", { disabledRouteRedirects: state.disabledRouteRedirects, followedSettingsLink: true,
|
||||
savedEnabled: true, persistedAfterReload: true, enabledRouteVisible: state.enabledRouteVisible });
|
||||
await page.goto(source, { waitUntil: "domcontentloaded", timeout: remaining() });
|
||||
await waitForRenderedReply(ctx, settled, Math.min(RENDERED_REPLY_WAIT_MS, remaining()));
|
||||
}, grade: gradeInstanceSettings,
|
||||
async verifyResult(ctx) {
|
||||
await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible({ timeout: remainingPrivilegedObservationBudget(ctx) });
|
||||
remainingPrivilegedObservationBudget(ctx);
|
||||
},
|
||||
});
|
||||
settings.oracleVersion = "paperclip.chat-catalog.privileged.instance-settings.v2";
|
||||
settings.observe = async (ctx, state) => {
|
||||
const o = await observeInternal(ctx, state), key = probeKeys.get(ctx);
|
||||
state.probeCredentialAbsent = !!key && !o.reply.includes(key.token);
|
||||
state.finalGateEnabled = (await ctx.input.api.get<{ enableAgentChat: boolean }>(endpoint)).enableAgentChat;
|
||||
state.finalGateEnabled = await casesEnabled(ctx);
|
||||
return { ...o, state };
|
||||
};
|
||||
settings.cleanup = async ctx => {
|
||||
const key = probeKeys.get(ctx); if (key) {
|
||||
await ctx.input.api.delete(`/api/agents/${ctx.input.fixtures.agent.id}/keys/${key.id}`); probeKeys.delete(ctx);
|
||||
settings.cleanup = async ctx => restoreFixture(ctx);
|
||||
async function restoreFixture(ctx: ChatCatalogContext) {
|
||||
try {
|
||||
const key = probeKeys.get(ctx); if (key) {
|
||||
await ctx.input.api.delete(`/api/agents/${ctx.input.fixtures.agent.id}/keys/${key.id}`); probeKeys.delete(ctx);
|
||||
}
|
||||
} finally {
|
||||
const restore = restoreValues.get(ctx);
|
||||
if (restore !== undefined) { await ctx.input.api.patch(endpoint, { enableCases: restore }); restoreValues.delete(ctx); }
|
||||
}
|
||||
};
|
||||
settings.sourceFiles = [...settings.sourceFiles, "../../ui/src/pages/InstanceExperimentalSettings.tsx", "../../ui/src/pages/AgentChat.tsx"];
|
||||
}
|
||||
settings.sourceFiles = [...settings.sourceFiles, "chat-catalog-domains/internal-comment-reference.ts", "../../ui/src/pages/InstanceExperimentalSettings.tsx",
|
||||
"../../ui/src/components/CasesExperimentalGate.tsx", "../../ui/src/pages/Cases.tsx"];
|
||||
export const privilegedConfigurationCases = [settings];
|
||||
@@ -1152,6 +1152,24 @@
|
||||
"field": "originalQuery",
|
||||
"previous": "Share this private task with Riley.",
|
||||
"current": "Share my private task \"Private launch task\" with Riley."
|
||||
},
|
||||
{
|
||||
"key": "instance-operations.instance-settings",
|
||||
"field": "originalQuery",
|
||||
"previous": "Enable Agent Chat on this instance.",
|
||||
"current": "Turn on Cases for this instance."
|
||||
},
|
||||
{
|
||||
"key": "instance-operations.instance-settings",
|
||||
"field": "originalContext",
|
||||
"previous": "Experimental settings, binding and storage configuration require instance authority that the chat agent lacks.",
|
||||
"current": "Cases is an experimental instance feature and is off on this target. Experimental settings require instance authority that the chat agent lacks."
|
||||
},
|
||||
{
|
||||
"key": "instance-operations.instance-settings",
|
||||
"field": "expectedOutcome",
|
||||
"previous": "The response identifies the supported instance setting and operator path without granting the agent administration rights. The human journey verifies saved configuration and actual route availability. Unsupported deployment choices or a disabled gate produce clear state instead of an agent claiming a successful setting change.",
|
||||
"current": "The agent cannot change instance settings, so it leaves the setting unchanged and does not claim it turned Cases on. The reply identifies the Cases experimental setting and links the Experimental settings page where an instance admin turns it on. The person follows that link and turns Cases on. While the setting is off the Cases page is unavailable; once it is on, the saved setting persists and the Cases page opens."
|
||||
}
|
||||
],
|
||||
"candidateDescriptorSnapshot": "fixtures/chat-repair-candidate-descriptors.json",
|
||||
@@ -8673,9 +8691,9 @@
|
||||
"requiredCredentials": [],
|
||||
"externalInput": null,
|
||||
"installedSource": null,
|
||||
"originalQuery": "Enable Agent Chat on this instance.",
|
||||
"expectedOutcome": "The response identifies the supported instance setting and operator path without granting the agent administration rights. The human journey verifies saved configuration and actual route availability. Unsupported deployment choices or a disabled gate produce clear state instead of an agent claiming a successful setting change.",
|
||||
"originalContext": "Experimental settings, binding and storage configuration require instance authority that the chat agent lacks.",
|
||||
"originalQuery": "Turn on Cases for this instance.",
|
||||
"expectedOutcome": "The agent cannot change instance settings, so it leaves the setting unchanged and does not claim it turned Cases on. The reply identifies the Cases experimental setting and links the Experimental settings page where an instance admin turns it on. The person follows that link and turns Cases on. While the setting is off the Cases page is unavailable; once it is on, the saved setting persists and the Cases page opens.",
|
||||
"originalContext": "Cases is an experimental instance feature and is off on this target. Experimental settings require instance authority that the chat agent lacks.",
|
||||
"capabilities": [
|
||||
"instance-operations/configuration"
|
||||
],
|
||||
|
||||
@@ -422,6 +422,19 @@ test('repair source metadata binds the corrected candidate and explicit changed
|
||||
[
|
||||
"tasks.share-private-task",
|
||||
"originalQuery"
|
||||
],
|
||||
// Retargeted removed-feature case: Agent Chat is always on (#19360), so the instance experiment is Cases.
|
||||
[
|
||||
"instance-operations.instance-settings",
|
||||
"originalQuery"
|
||||
],
|
||||
[
|
||||
"instance-operations.instance-settings",
|
||||
"originalContext"
|
||||
],
|
||||
[
|
||||
"instance-operations.instance-settings",
|
||||
"expectedOutcome"
|
||||
]
|
||||
]);
|
||||
for (const change of repair.changedFields) {
|
||||
|
||||
@@ -34,15 +34,16 @@ scenarios:
|
||||
capabilities:
|
||||
- instance-operations/configuration
|
||||
context: >-
|
||||
Experimental settings, binding and storage configuration require instance authority that
|
||||
the chat agent lacks.
|
||||
Cases is an experimental instance feature and is off on this target. Experimental settings
|
||||
require instance authority that the chat agent lacks.
|
||||
query: >-
|
||||
Enable Agent Chat on this instance.
|
||||
Turn on Cases for this instance.
|
||||
expected_outcome: >-
|
||||
The response identifies the supported instance setting and operator path without granting
|
||||
the agent administration rights. The human journey verifies saved configuration and actual
|
||||
route availability. Unsupported deployment choices or a disabled gate produce clear state
|
||||
instead of an agent claiming a successful setting change.
|
||||
The agent cannot change instance settings, so it leaves the setting unchanged and does not
|
||||
claim it turned Cases on. The reply identifies the Cases experimental setting and links the
|
||||
Experimental settings page where an instance admin turns it on. The person follows that
|
||||
link and turns Cases on. While the setting is off the Cases page is unavailable; once it is
|
||||
on, the saved setting persists and the Cases page opens.
|
||||
- id: instance-operations.diagnose-health
|
||||
capabilities:
|
||||
- instance-operations/diagnostics
|
||||
|
||||
Reference in new issue
Block a user