mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 18:56:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades saved product state > - A review of full cohort 3 found three cases whose oracle passed wrong behavior, and eight passes whose final evidence saved no rendered links > - `companies.agent-observability` passed although the agent denied the saved collector. `inbox-search.show-my-work` counted links that existed only in run-stream prose. `instance-operations.restart-service` accepted a reply whose only link was the `/<prefix>/` placeholder > - The final evidence read ran on a page that had not rendered the conversation yet, so it saved no links > - This pull request grades those cases on saved-state reads, the saved reply and resolved links, and reads final evidence from the rendered conversation > - The benefit is that these cases fail when the product is wrong and keep their links when it is right ## Linked Issues or Issue Description Refs #20998 Refs #20999 **What happened?** - `companies.agent-observability`: the board saved and tested a collector. Asked to acknowledge it, the agent said there was no endpoint or credentials. The oracle graded only human-side state, so it passed. - `inbox-search.show-my-work`: the saved reply said "Answered above" over nothing. The links the oracle counted came from run-stream prose that the bubble also shows. - `instance-operations.restart-service`: the reply's only action link was `/<prefix>/company/settings/instance`. The oracle did not check links. - Eight cohort-3 passes (`access.other-company`, `agents.agent-api-key`, `assistant-connections.revoke-personal-grant`, `budgets-costs.resolve-budget`, `recovery.retry-exhausted`, `steering.stop-versus-pause`, `task-coordination.pause-tree`, `teams.install-team`) saved a final observation with no links and a failed grade. The graded reload had the links. The final evidence read ran after `verifyResult` left the page mid-load, and an expired render wait from an earlier document made the read instant. **Expected behavior** Each oracle fails the observed wrong behavior and passes valid, reworded behavior. Final evidence holds the links that the conversation renders. **Steps to reproduce** Grade the retained cohort-3 observations with the old oracles. They pass. Grade them with these oracles. The three wrong cases fail. ## What Changed - `companies.agent-observability` (oracle v4): each acknowledgement turn must read the saved trace export status successfully. The read is bound to the company by its redaction-safe `call_api` receipt. A reply must also render a link to `/<prefix>/company/settings` for the fixture company. The reply's wording is not matched. - `inbox-search.show-my-work` (oracle v5): actionable links must be carried by the saved reply: a Markdown link target in the saved body, or a task identifier written in it, which the conversation links. New shared helper `savedReplyLinks`. - `instance-operations.restart-service` (oracle v3): new check `reply-links-resolve` rejects any rendered or saved link target that keeps a template placeholder (`<...>` or `{...}`). New helpers `markdownLinkTargets` and `unresolvedLinkTarget` in `chat-rendered-links.ts`. - Link capture: the final success-path evidence read reopens the rendered conversation first. An expired render wait now belongs to its loaded document, not to the Playwright page. - Updated `chat-catalog-descriptors.json` for the three oracle versions and source files. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/identity-observability.test.ts tests/runner-e2e/chat-catalog-domains/internal.test.ts tests/runner-e2e/chat-catalog-domains/privileged-oracles.test.ts tests/runner-e2e/chat-catalog-domains/privileged-restart-history.test.ts tests/runner-e2e/chat-rendered-links.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-catalog-fingerprint.test.ts` - Calibration tests for each case: a valid result, a reworded valid result, and the broken cohort-3 behavior. - Red/green: the new render-wait test fails on the old wait logic and passes with this change. - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - A paid run on a combined branch with the two product fixes is recorded in a comment on this pull request. ## Risks - `companies.agent-observability` needs the agent-readable status route from #20998. Without it, the case fails, which is correct. - The show-my-work check needs links in the saved reply. Before #20999, an OpenCode reply that left its answer in the run stream fails this case. - The final evidence read can wait up to 30 seconds to reopen the conversation. The read is still bounded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
158 lines
11 KiB
TypeScript
158 lines
11 KiB
TypeScript
import { collectConversationLinks, conversationLinkLocator, markdownLinkTargets, type RenderedConversationLink } from "../chat-rendered-links.js";
|
|
import { expect } from "@playwright/test";
|
|
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
import type { ChatRun } from "../chat-flow.js";
|
|
|
|
export interface Entity {
|
|
id: string; companyId?: string; name?: string; title?: string; description?: string | null;
|
|
identifier?: string; urlKey?: string; status?: string; priority?: string; assigneeAgentId?: string | null;
|
|
projectId?: string | null; parentId?: string | null; goalId?: string | null; goalIds?: string[];
|
|
level?: string; ownerAgentId?: string | null; archivedAt?: string | null;
|
|
key?: string; body?: string; latestRevisionId?: string; latestRevisionNumber?: number;
|
|
createdByAgentId?: string | null; updatedByAgentId?: string | null; createdByRunId?: string | null;
|
|
createdAt?: string; updatedAt?: string; authorAgentId?: string | null; authorUserId?: string | null;
|
|
issueId?: string; action?: string; agentId?: string | null; actorId?: string; actorType?: string;
|
|
entityId?: string; documentId?: string; originalRevisionId?: string | null; currentRevisionId?: string | null; resolvedByAgentId?: string | null; selectedText?: string; resolvedAt?: string | null; details?: Record<string, unknown>;
|
|
}
|
|
export interface InternalObservation {
|
|
scenarioId: string; origin: string; companyId: string; leadId: string; state: Record<string, unknown>;
|
|
tasks: Entity[]; projects: Entity[]; goals: Entity[]; comments: Entity[];
|
|
renderedLinkEvidence?: RenderedConversationLink[]; activity: Entity[]; annotations?: Entity[]; agents?: Entity[]; documents: Entity[]; revisions: Entity[]; reply: string; links: string[]; humanQueries: string[]; runs: ChatRun[];
|
|
}
|
|
export function entity(state: Record<string, unknown>, key: string): Entity {
|
|
const value = state[key];
|
|
if (!value || typeof value !== "object" || typeof (value as Entity).id !== "string") throw new Error(`Missing fixture ${key}`);
|
|
return value as Entity;
|
|
}
|
|
export const companyPath = (ctx: ChatCatalogContext) => `/api/companies/${ctx.input.fixtures.company.id}`;
|
|
export async function project(ctx: ChatCatalogContext, name = "Launch", extra: Record<string, unknown> = {}) {
|
|
return ctx.input.api.post<Entity>(`${companyPath(ctx)}/projects`, { name, ...extra });
|
|
}
|
|
export async function task(ctx: ChatCatalogContext, title: string, projectId: string, extra: Record<string, unknown> = {}) {
|
|
// Backlog fixtures never start provider work. These are prerequisite records, not requested outcomes.
|
|
return ctx.input.api.post<Entity>(`${companyPath(ctx)}/issues`, { title, projectId, status: "backlog", priority: "medium", ...extra });
|
|
}
|
|
export async function goal(ctx: ChatCatalogContext, title: string, extra: Record<string, unknown> = {}) {
|
|
return ctx.input.api.post<Entity>(`${companyPath(ctx)}/goals`, { title, level: "company", ownerAgentId: ctx.input.fixtures.agent.id, ...extra });
|
|
}
|
|
export async function putDocument(ctx: ChatCatalogContext, issueId: string, body: string, baseRevisionId: string | null = null) {
|
|
return ctx.input.api.putSensitive<Entity>(`/api/issues/${issueId}/documents/plan`, {
|
|
title: "Website implementation plan", format: "markdown", body, baseRevisionId,
|
|
});
|
|
}
|
|
export async function observeInternal(ctx: ChatCatalogContext, state: Record<string, unknown>): Promise<InternalObservation> {
|
|
await ctx.refreshIssue();
|
|
const { api, fixtures } = ctx.input;
|
|
const [tasks, projects, goals, runs] = await Promise.all([
|
|
api.getRunObservation<Entity[]>(`${companyPath(ctx)}/issues?includeAgentCreatedTasks=true&limit=1000`),
|
|
api.get<Entity[]>(`${companyPath(ctx)}/projects?includeArchived=true`),
|
|
api.get<Entity[]>(`${companyPath(ctx)}/goals`), ctx.allRuns(),
|
|
]);
|
|
const source = ctx.issue();
|
|
const chat = source ? await api.get<Entity[]>(`/api/issues/${source.id}/comments?order=asc`) : [];
|
|
const replies = chat.filter(row => row.authorAgentId === fixtures.agent.id && runs.some(run => run.id === row.createdByRunId));
|
|
const target = state.task ? entity(state, "task") : undefined;
|
|
const comments = target ? await api.get<Entity[]>(`/api/issues/${target.id}/comments?order=asc`) : [];
|
|
const companyActivity = await api.getRunObservation<Entity[]>(`${companyPath(ctx)}/activity?limit=1000`);
|
|
const taskActivity = target ? await api.get<Entity[]>(`/api/issues/${target.id}/activity`) : [];
|
|
const activity = [...new Map([...companyActivity, ...taskActivity].map(row => [row.id, row])).values()];
|
|
const renderedLinkEvidence = source ? await collectConversationLinks(ctx.input.page, {
|
|
origin: api.baseURL, companyId: fixtures.company.id, companyPrefix: fixtures.company.issuePrefix,
|
|
sourceId: source.id, leadId: fixtures.agent.id, comments: chat, runs, projects, activity,
|
|
}) : [];
|
|
const links = [...new Set(renderedLinkEvidence.map(link => link.href))];
|
|
const documents: Entity[] = []; const revisions: Entity[] = [];
|
|
if (target) {
|
|
for (const summary of await api.get<Entity[]>(`/api/issues/${target.id}/documents`)) {
|
|
const endpoint = `/api/issues/${target.id}/documents/${encodeURIComponent(summary.key!)}`;
|
|
documents.push(await api.get<Entity>(endpoint));
|
|
revisions.push(...await api.get<Entity[]>(`${endpoint}/revisions`));
|
|
}
|
|
}
|
|
return { scenarioId: ctx.scenario.id, origin: api.baseURL, companyId: fixtures.company.id, leadId: fixtures.agent.id,
|
|
state, tasks: tasks.filter(row => row.id !== source?.id), projects, goals, runs, comments, activity, documents, revisions,
|
|
renderedLinkEvidence, humanQueries: chat.filter(row => !row.authorAgentId).map(row => row.body ?? ""), reply: replies.map(row => row.body ?? "").join("\n"), links };
|
|
}
|
|
export function hasLink(o: InternalObservation, kind: "issues" | "projects" | "goals", row: Entity): boolean {
|
|
return o.links.some(href => {
|
|
try {
|
|
const url = new URL(href, o.origin);
|
|
if (url.origin !== new URL(o.origin).origin) return false;
|
|
const parts = url.pathname.split("/").map(decodeURIComponent); const index = parts.indexOf(kind);
|
|
const target = parts[index + 1];
|
|
return index >= 0 && !!target && [row.id, row.identifier, row.urlKey].filter(Boolean).includes(target);
|
|
} catch { return false; }
|
|
});
|
|
}
|
|
/**
|
|
* Rendered links that the saved reply itself carries: a Markdown link target in a saved reply body, or a
|
|
* task identifier written in that body, which the conversation links. A chat bubble can also render
|
|
* prose from the run stream that was never saved; links found only there are not part of the reply that
|
|
* history, notifications and other channels deliver. Only link structure is matched, never wording.
|
|
*/
|
|
export function savedReplyLinks(o: Pick<InternalObservation, "links" | "origin" | "reply">): string[] {
|
|
const location = (href: string) => {
|
|
try { const url = new URL(href, o.origin); return url.origin === new URL(o.origin).origin ? decodeURIComponent(url.pathname).replace(/\/+$/, "") + url.search : url.href; }
|
|
catch { return null; }
|
|
};
|
|
const targets = new Set(markdownLinkTargets(o.reply).map(location));
|
|
return o.links.filter(href => {
|
|
const at = location(href); if (!at) return false;
|
|
if (targets.has(at)) return true;
|
|
const parts = at.split("?")[0]!.split("/"); const identifier = parts[parts.indexOf("issues") + 1];
|
|
return parts.includes("issues") && !!identifier && /^[A-Za-z][A-Za-z0-9]*-\d+$/.test(identifier)
|
|
&& new RegExp(`(?<![\\w-])${identifier}(?![\\w-])`, "i").test(o.reply);
|
|
});
|
|
}
|
|
export function observation(value: unknown): InternalObservation | undefined {
|
|
if (!value || typeof value !== "object") return;
|
|
const o = value as InternalObservation;
|
|
if (!o.state || ![o.tasks, o.projects, o.goals, o.comments, o.activity, o.documents, o.revisions, o.links, o.humanQueries, o.runs].every(Array.isArray)
|
|
|| typeof o.reply !== "string" || typeof o.origin !== "string" || typeof o.companyId !== "string" || typeof o.leadId !== "string") return;
|
|
return o;
|
|
}
|
|
export function checks() {
|
|
const rows: Array<{ id: string; passed: boolean; detail: string }> = [];
|
|
return { add(id: string, passed: boolean, detail: string) { rows.push({ id, passed, detail }); },
|
|
result() { return { passed: rows.length > 0 && rows.every(row => row.passed), checks: rows }; } };
|
|
}
|
|
export function sameTask(before: Entity, after: Entity | undefined, except: readonly string[] = []) {
|
|
return !!after && ["id", "companyId", "title", "description", "projectId", "status", "priority", "assigneeAgentId", "goalId"]
|
|
.filter(key => !except.includes(key)).every(key => before[key as keyof Entity] === after[key as keyof Entity]);
|
|
}
|
|
|
|
/** Follow the exact rendered chat links, check the real result view, and return to the source. */
|
|
export async function verifyInternalResult(ctx: ChatCatalogContext, _state: Record<string, unknown>, value: unknown, remainingBudget?: () => number) {
|
|
const options = () => remainingBudget ? { timeout: remainingBudget() } : {};
|
|
const o = observation(value);
|
|
if (!o) throw new Error("Result navigation requires a complete observation");
|
|
const sourceUrl = ctx.input.page.url();
|
|
const candidates: Array<{ kind: "issues" | "projects" | "goals"; row: Entity }> = [
|
|
...o.tasks.map(row => ({ kind: "issues" as const, row })),
|
|
...o.projects.map(row => ({ kind: "projects" as const, row })),
|
|
...o.goals.map(row => ({ kind: "goals" as const, row })),
|
|
].filter(target => hasLink(o, target.kind, target.row));
|
|
if (!candidates.length) throw new Error("No independently matched result link was rendered");
|
|
for (const { kind, row } of candidates) {
|
|
const href = o.links.find(link => hasLink({ ...o, links: [link] }, kind, row))!;
|
|
const link = internalResultLink(ctx, o, href).first();
|
|
await expect(link).toBeVisible(options());
|
|
await link.click(options());
|
|
await ctx.input.page.waitForURL(url => {
|
|
const parts = url.pathname.split("/").map(decodeURIComponent); const at = parts.indexOf(kind);
|
|
return at >= 0 && [row.id, row.identifier, row.urlKey].includes(parts[at + 1]);
|
|
}, options());
|
|
await expect(ctx.input.page.getByRole("heading", { name: row.title ?? row.name!, exact: true }).first()).toBeVisible(options());
|
|
await ctx.input.evidence(`chat-catalog-opened-${kind}-${row.id}.json`, { id: row.id, kind, url: ctx.input.page.url(), title: row.title ?? row.name });
|
|
await ctx.input.page.goto(sourceUrl, { waitUntil: "domcontentloaded", ...options() });
|
|
await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible(options());
|
|
remainingBudget?.();
|
|
}
|
|
}
|
|
|
|
export function internalResultLink(ctx: ChatCatalogContext, o: InternalObservation, href: string) {
|
|
const receipt = o.renderedLinkEvidence?.find(link => link.href === href);
|
|
if (!receipt) throw new Error("Result navigation requires current-conversation rendered-link custody");
|
|
return conversationLinkLocator(ctx.input.page, receipt);
|
|
}
|