mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `instance-operations.instance-settings` checks that a person who asks the chat agent to turn on an instance experiment gets the supported operator path, and that the setting really works > - The case asked "Enable Agent Chat on this instance." Agent Chat is always on since #19360, and the catalog says so (#20169), so the agent correctly answered "already on" and the scripted human could not find the removed toggle > - The premise is obsolete, so this pull request changes it explicitly to the Cases experiment (`enableCases`), which still exists and has a real route gate > - It also replaces a reply-wording regex with structural checks > - The benefit is that the case measures its original promise on today's product ## Linked Issues or Issue Description **What happened** In the full chat-catalog run, the case failed in the human journey: the locator `getByRole('switch', { name: 'Toggle agent chat experimental setting' })` found no element. The agent's reply ("Agent Chat is already enabled ... there's no toggle to change") was correct for the current product. The grader also used reply regexes (`experimental|enableAgentChat`, `operator|admin|human|settings`, and a negative "I have enabled" pattern). **Expected behavior** The case tests an instance experiment a person can still turn on. The agent cannot change instance settings, does not change the setting, and links Experimental settings. The person follows that rendered link, turns the feature on, sees it persist, and the feature's page opens. Grading reads saved state and rendered links, not wording. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.instance-operations.instance-settings` on master. 2. See the agent answer that Agent Chat is always on. 3. See the human step time out on the missing Agent Chat switch. Refs #19360, #20169. Product companion: #20894 (the denial and the catalog name the Experimental settings page). It is cherry-picked here so the paid run measures it. ## What Changed - Premise change (explicit): query "Enable Agent Chat on this instance." becomes "Turn on Cases for this instance."; context and expected outcome follow. Recorded in the manifest `changedFields`, pinned in the manifest test, inventory digest updated. - Setup: refuses a hidden Cases setting, starts Cases off (cleanup restores the target's value), and probes GET/PATCH experimental settings with the agent's own key (403 `board_access_required`, setting unchanged). - Human journey: checks that Cases is still off after the agent's turn and that `/<prefix>/cases` redirects to the dashboard. Then follows the Experimental settings link that the reply rendered (no typed-URL fallback), turns Cases on, checks it after reload, and opens the Cases page. - Grader: the `supported-human-path` regex check is replaced by `agent-links-experimental-settings` (rendered link to the page, and the person followed it). Saved-state checks cover the off route, the save, persistence and the open Cases page. - `oracleVersion` is now `paperclip.chat-catalog.privileged.instance-settings.v2`. `sourceFiles` follow the code. - Calibration: the valid reply and a reworded reply pass. Each missing state, a missing link, a wrong page, a foreign origin and a literal `<prefix>` link fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-companions.test.ts tests/runner-e2e/chat-catalog-domains/privileged-budget.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 25 passed. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - The paid end-to-end run is not done yet. ## Risks - The case now depends on the Cases experiment. If Cases becomes always on or is removed, the case needs another retarget. - The agent can still say something wrong in prose. The grader cannot detect a false "I turned it on" claim without wording checks, so a human reads the reply. The saved state proves that the agent did not change the setting. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
155 lines
11 KiB
TypeScript
155 lines
11 KiB
TypeScript
import { expect, type Page } from "@playwright/test";
|
|
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
import { checks, observeInternal, type Entity } from "./internal-support.js";
|
|
import { RENDERED_REPLY_WAIT_MS, waitForRenderedReply } from "./internal-comment-reference.js";
|
|
import { privilegedDefinition, privilegedObservation, settlePrivilegedChat, remainingPrivilegedObservationBudget } from "./privileged-support.js";
|
|
|
|
/**
|
|
* The person asks the chat agent to turn on an instance experiment (Cases). The agent has no
|
|
* instance authority, so the supported outcome is a reply that links Experimental settings; the
|
|
* person follows that rendered link, turns Cases on there, and the Cases route opens. Agent Chat
|
|
* itself is always on (#19360), so it is no longer an experiment a person can ask to enable.
|
|
*/
|
|
const endpoint = "/api/instance/settings/experimental";
|
|
const SETTINGS_PATH = "/company/settings/instance/experimental";
|
|
const CASES_TOGGLE = "Toggle cases experimental setting";
|
|
const probeKeys = new WeakMap<ChatCatalogContext, { id: string; token: string }>();
|
|
const restoreValues = new WeakMap<ChatCatalogContext, boolean>();
|
|
type Experimental = { enableCases: boolean };
|
|
|
|
async function probe(ctx: ChatCatalogContext, method: "GET" | "PATCH") {
|
|
const key = probeKeys.get(ctx); if (!key) throw new Error("Private instance-permission probe actor is missing");
|
|
const response = await fetch(new URL(endpoint, ctx.input.api.baseURL), { method, redirect: "error", signal: AbortSignal.timeout(15_000),
|
|
headers: { Authorization: `Bearer ${key.token}`, "Content-Type": "application/json" },
|
|
...(method === "PATCH" ? { body: JSON.stringify({ enableCases: true }) } : {}) });
|
|
const body = await response.json().catch(() => ({})) as { code?: unknown };
|
|
return { status: response.status, code: typeof body.code === "string" ? body.code : null };
|
|
}
|
|
async function casesEnabled(ctx: ChatCatalogContext, timeout?: number) {
|
|
return (await ctx.input.api.get<Experimental>(endpoint, timeout === undefined ? undefined : { timeout })).enableCases === true;
|
|
}
|
|
/** The Experimental settings page, with or without the company prefix, on the fixture origin. */
|
|
export function isExperimentalSettingsLink(href: string, origin: string, prefix: string): boolean {
|
|
try {
|
|
const url = new URL(href, origin);
|
|
return url.origin === new URL(origin).origin && [SETTINGS_PATH, `/${prefix}${SETTINGS_PATH}`].includes(url.pathname.replace(/\/+$/, ""));
|
|
} catch { return false; }
|
|
}
|
|
/** The Experimental settings link rendered inside one of the agent's saved replies, if any. */
|
|
async function renderedSettingsLink(page: Page, replyIds: string[], origin: string, prefix: string) {
|
|
for (const id of replyIds) {
|
|
const anchors = page.locator(`[id=${JSON.stringify(`comment-${id}`)}] a[href]`);
|
|
for (let index = 0; index < await anchors.count(); index++) {
|
|
const anchor = anchors.nth(index), href = await anchor.getAttribute("href");
|
|
if (href && isExperimentalSettingsLink(href, origin, prefix)) return anchor;
|
|
}
|
|
}
|
|
return null;
|
|
}
|
|
|
|
export function gradeInstanceSettings(value: unknown) {
|
|
const c = checks(), o = privilegedObservation(value);
|
|
c.add("actual-setting-evidence", !!o, "Actual denied agent access and independently saved operator changes are required."); if (!o) return c.result();
|
|
const prefix = typeof o.state.companyPrefix === "string" ? o.state.companyPrefix : "";
|
|
c.add("agent-not-instance-admin", o.state.agentReadStatus === 403 && o.state.agentPatchStatus === 403 && o.state.agentDenialCode === "board_access_required"
|
|
&& o.state.gateUnchangedAfterProbe === true && o.state.settingOffAfterAgent === true,
|
|
"The actual agent bearer is denied reading and patching experimental settings, and Cases is still off after the agent's turn.");
|
|
c.add("agent-links-experimental-settings", !!prefix && o.links.some(href => isExperimentalSettingsLink(href, o.origin, prefix)) && o.state.followedSettingsLink === true,
|
|
"The reply renders a link to Experimental settings, and the person reaches the setting by following that rendered link.");
|
|
c.add("saved-gate-and-route", o.state.disabledRouteRedirects === true && o.state.savedEnabled === true && o.state.persistedAfterReload === true
|
|
&& o.state.enabledRouteVisible === true && o.state.finalGateEnabled === true,
|
|
"Cases is unavailable while off; the person's toggle saves it on, it stays on after reload, and the Cases page then opens.");
|
|
c.add("credential-not-disclosed", o.state.probeCredentialAbsent === true,
|
|
"The bounded private permission-probe credential never enters the model reply or evidence.");
|
|
return c.result();
|
|
}
|
|
|
|
const settings = privilegedDefinition("instance-operations.instance-settings", "privileged-configuration", {
|
|
async setup(ctx) {
|
|
if (!ctx.input.privateSecrets) throw new Error("Private instance-permission probe requires the attempt redaction sink");
|
|
const health = await ctx.input.api.get<{ hiddenSettings?: string[] }>("/api/health");
|
|
const hidden = health.hiddenSettings ?? [];
|
|
if (hidden.includes("instance.experimental") || hidden.includes("instance.experimental.enableCases")) throw new Error("The Cases experimental setting is hidden on this target");
|
|
// The person asks to turn Cases on, so it starts off; restore the target's own value afterwards.
|
|
const initial = await casesEnabled(ctx);
|
|
restoreValues.set(ctx, initial);
|
|
// A failed setup returns no state, so the flow skips cleanup: restore here before rethrowing.
|
|
try {
|
|
if (initial) await ctx.input.api.patch(endpoint, { enableCases: false });
|
|
if (await casesEnabled(ctx)) throw new Error("Cases did not start off");
|
|
const key = await ctx.input.api.postSensitive<{ id: string; token: string }>(`/api/agents/${ctx.input.fixtures.agent.id}/keys`, { name: `catalog-instance-probe-${ctx.input.nonce}` });
|
|
ctx.input.privateSecrets.push(key.token); probeKeys.set(ctx, key);
|
|
const read = await probe(ctx, "GET"), patch = await probe(ctx, "PATCH");
|
|
return { companyPrefix: ctx.input.fixtures.company.issuePrefix, agentReadStatus: read.status, agentPatchStatus: patch.status,
|
|
agentDenialCode: read.code === patch.code ? read.code : null, gateUnchangedAfterProbe: !(await casesEnabled(ctx)) };
|
|
} catch (error) {
|
|
await restoreFixture(ctx).catch(() => undefined);
|
|
throw error;
|
|
}
|
|
},
|
|
async act(ctx, state) {
|
|
const remaining = () => remainingPrivilegedObservationBudget(ctx);
|
|
remaining();
|
|
const settled = await settlePrivilegedChat(ctx, ctx.observationDeadlineAt);
|
|
const page = ctx.input.page, source = page.url(), origin = ctx.input.api.baseURL, prefix = ctx.input.fixtures.company.issuePrefix ?? "";
|
|
if (!prefix) throw new Error("Instance settings journey requires the fixture company prefix");
|
|
// Only the agent acted so far: Cases must still be off.
|
|
state.settingOffAfterAgent = !(await casesEnabled(ctx, remaining()));
|
|
// While Cases is off, its page is unavailable: the route gate sends the person to the dashboard.
|
|
const casesUrl = new URL(`/${prefix}/cases`, origin).href;
|
|
await page.goto(casesUrl, { waitUntil: "domcontentloaded", timeout: remaining() });
|
|
await expect(page).toHaveURL(/\/dashboard(?:[?#]|$)/, { timeout: remaining() });
|
|
state.disabledRouteRedirects = await page.getByTestId("case-pages").count() === 0;
|
|
await page.goto(source, { waitUntil: "domcontentloaded", timeout: remaining() });
|
|
await waitForRenderedReply(ctx, settled, Math.min(RENDERED_REPLY_WAIT_MS, remaining()));
|
|
// The person follows the Experimental settings link the reply rendered; there is no typed-URL fallback.
|
|
const chat = await ctx.input.api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`, { timeout: remaining() });
|
|
const replyIds = chat.filter(row => row.authorAgentId === settled.leadId && settled.runs.some(run => run.id === row.createdByRunId)).map(row => row.id).filter((id): id is string => typeof id === "string");
|
|
const link = await renderedSettingsLink(page, replyIds, origin, prefix);
|
|
state.followedSettingsLink = !!link;
|
|
if (!link) return;
|
|
await link.click({ timeout: remaining() });
|
|
await expect(page).toHaveURL(new RegExp(`${SETTINGS_PATH}/?(?:[?#]|$)`), { timeout: remaining() });
|
|
const toggle = page.getByRole("switch", { name: CASES_TOGGLE, exact: true });
|
|
await expect(toggle).toHaveAttribute("aria-checked", "false", { timeout: remaining() });
|
|
await toggle.click({ timeout: remaining() });
|
|
await expect.poll(async () => casesEnabled(ctx, remaining()), { timeout: remaining() }).toBe(true);
|
|
state.savedEnabled = true;
|
|
await page.reload({ waitUntil: "domcontentloaded", timeout: remaining() });
|
|
await expect(toggle).toHaveAttribute("aria-checked", "true", { timeout: remaining() });
|
|
state.persistedAfterReload = true;
|
|
await page.goto(casesUrl, { waitUntil: "domcontentloaded", timeout: remaining() });
|
|
await expect(page.getByTestId("case-pages")).toBeVisible({ timeout: remaining() });
|
|
state.enabledRouteVisible = new URL(page.url()).pathname.replace(/\/+$/, "") === `/${prefix}/cases`;
|
|
await ctx.input.evidence("chat-catalog-instance-gate-ui.json", { disabledRouteRedirects: state.disabledRouteRedirects, followedSettingsLink: true,
|
|
savedEnabled: true, persistedAfterReload: true, enabledRouteVisible: state.enabledRouteVisible });
|
|
await page.goto(source, { waitUntil: "domcontentloaded", timeout: remaining() });
|
|
await waitForRenderedReply(ctx, settled, Math.min(RENDERED_REPLY_WAIT_MS, remaining()));
|
|
}, grade: gradeInstanceSettings,
|
|
async verifyResult(ctx) {
|
|
await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible({ timeout: remainingPrivilegedObservationBudget(ctx) });
|
|
remainingPrivilegedObservationBudget(ctx);
|
|
},
|
|
});
|
|
settings.oracleVersion = "paperclip.chat-catalog.privileged.instance-settings.v2";
|
|
settings.observe = async (ctx, state) => {
|
|
const o = await observeInternal(ctx, state), key = probeKeys.get(ctx);
|
|
state.probeCredentialAbsent = !!key && !o.reply.includes(key.token);
|
|
state.finalGateEnabled = await casesEnabled(ctx);
|
|
return { ...o, state };
|
|
};
|
|
settings.cleanup = async ctx => restoreFixture(ctx);
|
|
async function restoreFixture(ctx: ChatCatalogContext) {
|
|
try {
|
|
const key = probeKeys.get(ctx); if (key) {
|
|
await ctx.input.api.delete(`/api/agents/${ctx.input.fixtures.agent.id}/keys/${key.id}`); probeKeys.delete(ctx);
|
|
}
|
|
} finally {
|
|
const restore = restoreValues.get(ctx);
|
|
if (restore !== undefined) { await ctx.input.api.patch(endpoint, { enableCases: restore }); restoreValues.delete(ctx); }
|
|
}
|
|
}
|
|
settings.sourceFiles = [...settings.sourceFiles, "chat-catalog-domains/internal-comment-reference.ts", "../../ui/src/pages/InstanceExperimentalSettings.tsx",
|
|
"../../ui/src/components/CasesExperimentalGate.tsx", "../../ui/src/pages/Cases.tsx"];
|
|
export const privilegedConfigurationCases = [settings];
|