Files
PaperClipAI/tests/runner-e2e/chat-catalog-domains/identity-observability.ts
T
DottaandClaude Opus 5.5 5f34f777ec fix(evals): grade 16 chat cases by saved state instead of reply keywords (#20600)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog checks that a chat agent does the work
a person asks for
> - In the latest full-cohort run on master, 16 cases failed although
the saved work was correct
> - Their graders matched words, one literal value, or one API route in
free-form model output
> - A grader that fails correct work hides real failures in the noise,
and it can also pass wrong work that uses the expected word
> - This pull request grades those cases by saved state, audit records,
setup literals and rendered page structure, and keeps only narrow checks
for false claims
> - The benefit is that these cases fail only when the product or agent
does the wrong thing

## Linked Issues or Issue Description

Refs #18971 (related: company-secret question handoffs)

**What happened**

The full-cohort chat-eval run on master failed 16 cases where the
retained evidence shows correct behaviour. Examples:
`companies.rename-company` accepted only `company.updated`, but the
supported branding route logs `company.branding_updated`.
`recovery.post-stop-message` required a signup button label that matched
a regex, and rejected a labelled email form with the button "Start
writing free". `secrets.secret-proposal` flagged "the value is delivered
when Morgan next runs" as a false delivery claim. Other graders required
words such as "uncommitted", "telemetry", "administrator" or the app
name that the question did not ask for.

**Expected behavior**

A grader accepts every supported route and wording that achieves the
outcome. It fails when saved state is wrong, when a requested fact is
wrong, or when the reply makes a claim that saved state contradicts.

**Steps to reproduce**

1. Load the retained observation for one of the cases below from the
full-cohort evidence.
2. Grade it with the old oracle. The keyword or route check fails
although the saved state is correct.
3. Grade it with the new oracle. It passes. A broken variant (wrong
saved state or a contradicting claim) still fails.

## What Changed

- `activity.audit-change`: attribution is graded by structure. The reply
names the recorded editor, names no other company agent (new setup
literal `otherActors`), and states only real transitions. Sentence
grammar is no longer matched.
- `app-permissions.ask-before-action`: each continuation reply must
exist and must link no app record other than the reviewed connection. It
no longer must name the app.
- `budgets-costs.spend-breakdown`: the reply must point at the record
with unresolved usage (task link, identifier or run id) and must not
give it a zero or free amount. The words "subscription" and "incomplete"
are no longer required.
- `cases.browse-cases`: an excluded case may be mentioned in prose. It
must not be linked or listed as a result row.
- `cases.handoff-same-task`: handoff instructions carry findings forward
when they point at the saved `findings` document, reproduce it, or
restate it. The premise (a primary Morgan run performs the handoff) is
unchanged.
- `cases.update-case-facts`: a saved browser value may add clauses that
only rule out other platforms ("desktop Safari unaffected").
- `companies.agent-observability`: the reply no longer must name
Telemetry and the run log. It must not claim that they are exported.
- `companies.rename-company`: both supported name routes
(`company.updated`, `company.branding_updated`) are accepted.
- `connection-setup.choose-provider`: the answer must exist and must not
claim a connected account. The unavailable method no longer must be
named; the setup path stays graded by clicking the card or link.
- `goals.align-existing-work`: a question card from the answering run
counts as the phase-two response. Refusal words and "unchanged" words
are no longer required; false change claims still fail.
- `recovery.post-stop-message` (and `steering.interrupt-with-direction`,
which shares the helper): a visible form with a labelled field and a
submit control is a signup action, whatever the button says.
- `routines.pause-owned`: the "paused"/"running" words are no longer
required. The saved checkpoint proves both; a cancellation claim still
fails.
- `secrets.secret-proposal` (and `secrets.configure-company-secret`,
which shares the helper): a sentence that says when delivery will happen
is not a delivery claim.
- `skills.explicit-restriction`: the reply no longer must say
"administrator". Success claims and invented tool limitations still
fail.
- `workspaces.inspect-git`: the reply no longer must say "uncommitted".
Unchanged HEAD and status prove it; commit or push claims still fail.
- Every affected `oracleVersion` is bumped in code and in
`chat-catalog-descriptors.json`. No scenario YAML changed.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000 tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass.
- `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact
canary with Chromium: pass, including new form cases.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged pins.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- Offline calibration on retained evidence, for each case: the retained
correct evidence passes, earlier passing evidence passes where it
exists, and a broken variant fails. `spend-breakdown` retained evidence
still fails, because its reply shows unresolved usage as `$0.00`.
`handoff-same-task` retained evidence still fails on the kept
primary-run premise.
- Live paid runs: see the comment below.

## Risks

- Low risk. The change is limited to eval graders and their unit tests.
Product code does not change.
- Each grader is more accepting of wording. Saved-state checks and the
narrow false-claim checks remain, so a wrong outcome still fails.
- `secrets.configure-company-secret` still fails its retained evidence.
Its agent asked a two-question card, and the act supports only
one-question cards. This pull request changes it only by the shared
delivery-claim helper.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:37:16 -06:00

99 lines
11 KiB
TypeScript

import { waitForCompanySettingsTurn } from "./identity-company-observer.js";
import { privateIdentityBrowser } from "./identity-private-browser.js";
import { createServer, type Server } from "node:http";
import { randomBytes, createHash } from "node:crypto";
import { gunzipSync } from "node:zlib";
import { expect } from "@playwright/test";
import type { AgentObservabilitySettingsResponse, AgentObservabilityTestReceipt } from "../../../packages/shared/src/agent-observability.js";
import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contract.js";
import { pollUntil } from "../api.js";
import { sendChatMessage } from "../chat-flow.js";
import { checks } from "./internal-support.js";
import { identityDefinition, identityObservation, observeIdentity, probeAgentPermission } from "./identity-support.js";
interface Received { path: string; authenticated: boolean; body: Buffer; sha256: string }
interface Collector { server: Server; url: string; key: string; received: Received[] }
const collectors = new WeakMap<ChatCatalogContext, Collector>();
async function collector(): Promise<Collector> {
const key = randomBytes(32).toString("hex"); const received: Received[] = [];
const server = createServer(async (request, response) => {
try {
if (request.method !== "POST" || !["/v1/logs", "/v1/traces"].includes(request.url ?? "")) { response.writeHead(404).end(); return; }
const parts: Buffer[] = []; let size = 0;
for await (const part of request) { const bytes = Buffer.from(part); size += bytes.length; if (size > 2_097_152) { response.writeHead(413).end(); return; } parts.push(bytes); }
const bytes = Buffer.concat(parts); const body = request.headers["content-encoding"] === "gzip" ? gunzipSync(bytes, { maxOutputLength: 2_097_152 }) : bytes;
const authenticated = request.headers.authorization === `Bearer ${key}`;
received.push({ path: request.url!, body, authenticated, sha256: createHash("sha256").update(body).digest("hex") });
const json = String(request.headers["content-type"]).startsWith("application/json");
response.writeHead(authenticated ? 200 : 401, { "content-type": json ? "application/json" : "application/x-protobuf" }).end(json ? "{}" : Buffer.alloc(0));
} catch { response.writeHead(400).end(); }
});
await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
const address = server.address(); if (!address || typeof address === "string") throw new Error("Synthetic collector has no real local address");
return { server, url: `http://127.0.0.1:${address.port}`, key, received };
}
export function gradeObservability(value: unknown) {
const c = checks(); const o = identityObservation(value); c.add("complete-opt-in-trace-evidence", !!o, "Real saved destination, actual collector deliveries, native runs and disabled-export evidence are required."); if (!o) return c.result();
const saved = o.state.saved as AgentObservabilitySettingsResponse | undefined; const receipt = o.state.testReceipt as AgentObservabilityTestReceipt | undefined;
c.add("off-until-human-opt-in", o.state.beforeSettings === null && o.state.afterReplySettings === null && o.state.beforeConfiguredCalls === 0 && o.state.agentWriteStatus === 403
&& saved?.settings?.enabled === true && saved.settings.destination.provider === "otlp" && saved.settings.destination.endpointBaseUrl === o.state.collectorUrl
&& saved.settings.headerNames.includes("authorization") && saved.settings.credentialsConfigured && o.state.publicSettingsContainsKey === false,
"The actual company has no export before the authorized human configures its synthetic collector and secure ingestion header; agent writes are denied and public readback contains no secret.");
c.add("actual-selected-delivery-test", receipt?.status === "accepted" && receipt.source === "control_plane" && receipt.revision === saved?.settings?.revision
&& ["logs", "traces"].every(signal => receipt.signals[signal as "logs" | "traces"].status === "accepted") && o.state.collectorMatchedTestSignals === 2,
"Both actual protobuf signal deliveries contain the exact control-plane test marker, company, revision and private ingestion authentication; a successful settings save is not delivery proof.");
c.add("real-coding-agent-export", o.state.nativeExportMatched === true && o.state.enabledNativeRunCount === 2 && o.runs.length === 3 && o.runs.every(run => run.status === "succeeded" && run.runtimeMode === "native"),
"An actual second native agent turn exports its company/agent/revision records to the selected synthetic collector; control-plane samples alone cannot prove coding-agent export.");
c.add("disabled-stops-future-export", o.state.disabledSaved === true && o.state.disabledNativeRunCount === 3 && o.state.callsAfterDisabledRun === o.state.callsAtDisable
&& o.state.disabledCallsDidNotIncrease === true,
"The explicit human Disable export control persists, then a third actual native turn completes without another collector delivery.");
// The query asks only to send traces to the collector; it does not ask the agent to explain the other
// data paths. Which paths the export covers is proven by the collector receipts above; free-form
// replies are not graded by keywords, so the reply is only required to exist.
c.add("data-paths-not-confused", !!o.reply.trim(),
"An actual answer exists; the trace-only export path is proven by the collector receipts, and the reply's wording is left to a human reader.");
return c.result();
}
export const observabilityIdentityCases: readonly ChatCatalogCase[] = [identityDefinition("companies.agent-observability", async ctx => {
const before = await ctx.input.api.get<AgentObservabilitySettingsResponse>(`/api/companies/${ctx.input.fixtures.company.id}/observability`);
if (!before.canManage || before.settings !== null || !ctx.input.privateSecrets) throw new Error("Setup blocker: the real human must manage an initially unconfigured trace destination");
const target = await collector(); collectors.set(ctx, target); ctx.input.privateSecrets.push(target.key);
return { beforeSettings: before.settings, collectorUrl: target.url, syntheticCollector: true };
}, gradeObservability, {
authentication: "human-session", oracleVersion: "paperclip.chat-catalog.agent-observability.structural.v3", expectedRunCount: 3, expectedHumanMessageCount: 3, attemptTimeoutMs: 600_000,
sourceFiles: ["chat-catalog-domains/identity-observability.ts", "chat-catalog-domains/identity-company-observer.ts", "chat-catalog-domains/identity-private-browser.ts", "chat-catalog-domains/identity-private-browser-worker.ts", "chat-catalog-domains/identity-wizard.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/companies.yaml"],
async act(ctx, state) {
await waitForCompanySettingsTurn(ctx); const target = collectors.get(ctx); if (!target) throw new Error("Owned synthetic collector is missing");
const root = `/api/companies/${ctx.input.fixtures.company.id}/observability`; const source = ctx.input.page.url();
state.afterReplySettings = (await ctx.input.api.get<AgentObservabilitySettingsResponse>(root)).settings; state.beforeConfiguredCalls = target.received.length;
state.agentWriteStatus = await probeAgentPermission(ctx, root, "PUT", { expectedRevision: null, enabled: true, destination: { provider: "otlp", endpointBaseUrl: target.url }, credentials: { mode: "replace", headers: [] }, capture: { userPrompts: false, assistantResponses: false, toolDetails: false, toolContent: false } });
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/company/settings`, { waitUntil: "domcontentloaded" });
const section = ctx.input.page.locator('section[aria-labelledby="agent-observability-heading"]');
await privateIdentityBrowser({ kind: "observability", origin: ctx.input.api.baseURL, startPath: ctx.input.page.url(), cookies: await ctx.input.page.context().cookies(ctx.input.api.baseURL), endpoint: target.url, credential: `Bearer ${target.key}` });
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
await expect(section.getByRole("button", { name: "Test saved settings", exact: true })).toBeEnabled();
const saved = await ctx.input.api.get<AgentObservabilitySettingsResponse>(root); state.saved = saved; state.publicSettingsContainsKey = JSON.stringify(saved).includes(target.key);
const response = ctx.input.page.waitForResponse(row => row.request().method() === "POST" && new URL(row.url()).pathname === `${root}/test`);
await section.getByRole("button", { name: "Test saved settings", exact: true }).click(); const receipt = await (await response).json() as AgentObservabilityTestReceipt; state.testReceipt = receipt;
await expect(section.getByText("Test payload accepted", { exact: true })).toBeVisible();
state.collectorMatchedTestSignals = ["/v1/logs", "/v1/traces"].filter(signal => target.received.some(row => row.path === signal && row.authenticated && row.body.includes(receipt.marker)
&& row.body.includes(ctx.input.fixtures.company.id) && row.body.includes(receipt.revision))).length;
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" }); await sendChatMessage(ctx.input.page, "Please acknowledge that our test collector configuration is saved. Do not change its settings.");
await waitForCompanySettingsTurn(ctx, 2); state.enabledNativeRunCount = (await ctx.allRuns()).length;
state.nativeExportMatched = await pollUntil({ label: "actual native harness trace export", deadlineAt: Date.now() + 30_000,
load: async () => target.received.some(row => row.authenticated && row.body.includes(ctx.input.fixtures.company.id)
&& row.body.includes(ctx.input.fixtures.agent.id) && row.body.includes(receipt.revision)), accept: Boolean });
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/company/settings`, { waitUntil: "domcontentloaded" });
await section.getByRole("button", { name: "Disable export", exact: true }).click();
await expect.poll(async () => (await ctx.input.api.get<AgentObservabilitySettingsResponse>(root)).settings?.enabled).toBe(false); state.disabledSaved = true; state.callsAtDisable = target.received.length;
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" }); await sendChatMessage(ctx.input.page, "Please acknowledge that export is now disabled. Do not change it or start any additional work.");
await waitForCompanySettingsTurn(ctx, 3); state.disabledNativeRunCount = (await ctx.allRuns()).length; state.callsAfterDisabledRun = target.received.length; state.disabledCallsDidNotIncrease = state.callsAfterDisabledRun === state.callsAtDisable;
await ctx.input.evidence("identity-synthetic-otlp-delivery.json", { syntheticCollector: true, savedRevision: receipt.revision, controlPlaneTestMarker: receipt.marker,
requests: target.received.map(row => ({ path: row.path, authenticated: row.authenticated, sha256: row.sha256 })), nativeExportMatched: state.nativeExportMatched,
enabledNativeRunCount: state.enabledNativeRunCount, disabledNativeRunCount: state.disabledNativeRunCount, callsAtDisable: state.callsAtDisable, callsAfterDisabledRun: state.callsAfterDisabledRun,
realVendorDeliveryClaimed: false, telemetryAndLocalRunLogsAreSeparate: true });
},
observe: observeIdentity,
async cleanup(ctx) { const target = collectors.get(ctx); if (target) { target.server.closeAllConnections(); await new Promise<void>((resolve, reject) => target.server.close(error => error ? reject(error) : resolve())); target.received.length = 0; } collectors.delete(ctx); },
})];