Files
DottaandClaude Opus 5.5 819f61f1fd fix(evals): repair four cohort-15 chat cases and prefix board links in completion feedback (#21682)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys. It grades
saved state, rendered links and run ledgers.
> - Chat-eval cohort 15 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `36101ebcb9`)
failed four cases where the agent did the right thing or where our own
server text misled it.
> - One failure was a harness race introduced by #21667. Two failures
came from graders that read free-form text with keyword regexes. One
failure came from an unprefixed approval link in the server's completion
feedback, plus a grader that required a literal prefix.
> - This pull request fixes the harness step, replaces the free-form
graders with saved-state and clicked-link checks, and makes the server
emit company-prefixed board links.
> - The benefit is that these cases measure the feature again. The
retained correct outcomes pass, and the relevant broken outcomes still
fail.

## Linked Issues or Issue Description

Refs #21667

**What happened?**
In chat-eval cohort 15, four cases failed:
- `projects.share-project`: the harness followed the agent's ID-based
project link. The app then rewrote the URL to the project slug and
redrew the page. The owner step timed out because Riley's option in the
people picker was "detached from the DOM". #21667 introduced this path.
- `skills.author-skill`: the saved skill was a correct, richer
checklist. Per-sentence keyword regexes rejected it. For example,
"ignore the mouse" tripped the forbidden word "ignore".
- `inbox-search.show-decisions`: the reply linked the right approval and
tasks and said the question was "addressed to a different teammate". A
regex required the word "Riley".
- `agents.hire-needs-approval`: both replies linked `/approvals/<id>`
without the company prefix. The agent copied that link from
`server/src/services/native-runtime/native-completion-feedback.ts`. The
grader required a literal `/<prefix>/approvals/<id>`, although the board
redirects the unprefixed route.

**Expected behavior**
Correct agent behaviour passes. Grading uses saved state, setup literals
and rendered or clicked links, not free-form wording. The server gives
agents canonical company-prefixed board links.

**Steps to reproduce**
1. Use the retained cohort-15 evidence for the four cases.
2. Run the old graders on the retained skill markdown and decisions
reply. Both fail.
3. For share-project, open a project configuration page by ID while the
app rewrites the route after a delay, then use the access picker before
the rewrite completes.
4. For hire-needs-approval, finish a task with a pending linked
approval. The feedback links `/approvals/<id>`.

**Paperclip version or commit**
master `36101ebcb9` (cohort 15); fixed on top of current master.

## What Changed

- `projects.share-project` (`internal-privacy.ts`,
`runtime-support.ts`): after the harness follows the agent's link, it
waits until the URL settles on the project's canonical route key. This
reuses the #21667 approach for agent profiles: `settleAgentProfileRoute`
now calls a shared `settleRouteKey`, and `projectRouteKey` follows the
app's `projectRouteRef`. The owner's picker selection
(`addProjectAccessMember`) now retries as one unit. If the dialog or
picker closes, or an option detaches, it reopens them, searches again
and selects again. It continues only when the picker shows the person
and Add is enabled.
- `skills.author-skill` (`integrator-skills.ts`): removed the
keyword-regex checklist grader, the waiver regexes, and the reply
regexes for "global activation" and "attached to me" claims. The
saved-state checks stay: one agent-created writable skill, its current
revision authored by the responding agent, the same bytes on the skill
and the revision, the Skills/Studio inspection with reload, and
assignment scope from saved assignments and audit rows. The checklist
check now requires the setup literals `375`, `WCAG` and `AA` as whole
tokens in the saved SKILL.md and in the rendered editor. Whether the
skill still asks for keyboard-only navigation and visible labels, and
whether it waives a check, is left to human review. Oracle version
`skill-authoring.v10`.
- `inbox-search.show-decisions` (`internal-decision-search.ts`): removed
all reply regexes, including the `Riley` requirement and the
wording-based separation checks. The grade now uses saved state (pending
decisions unchanged, answered history unchanged, each person's own
attention queue and resolver) and rendered links to the exact approval,
the question's task and the history task. Oracle version
`decision-search.v3`.
- Product (`native-completion-feedback.ts`): the completion feedback now
links `/<prefix>/approvals/<id>` and `/<prefix>/issues/<identifier>`.
This fixes the approval link and the four sibling task links in the same
file.
- `agents.hire-needs-approval` (`runtime-hiring.ts`): the harness now
clicks each reply's rendered link to its approval and records where the
UI lands. The grade accepts a prefixed link, or an unprefixed link that
the board redirect lands on `/<prefix>/approvals/<id>` with the proposed
hire visible. Without such a link, the harness still opens the approval
directly so later phases keep their evidence, and the grade fails.
Oracle version `runtime-hiring.v4`.
- No eval YAML changed, so the TestOps manifest does not change.
`chat-catalog-descriptors.json` records the three oracle version bumps.

## Verification

- Red/green tests (each new test fails on the old code and passes on the
new code):
- `internal-privacy-share-route.test.ts` (real Chromium): the project
page rewrites its route after 0, 400 and 1500 ms, and one case closes
the picker while its first results render. With the old flow, the 400 ms
rewrite and the closed picker both time out on the option click (the
cohort-15 failure). With the fix, Riley is added on the settled route in
all cases.
- `integrator-skills.test.ts`: the retained cohort-15 SKILL.md passes.
The old grader rejects it. A stale skill body, another author, and a
lost 375 width still fail.
- `internal-decision-search.test.ts`: the retained cohort-15 reply and a
reworded reply pass. The old grader rejects the retained reply. A
missing or wrong approval, question-task or history-task link fails.
- `native-completion-feedback.test.ts`: a pending linked approval
produces `[the pending approval](/<prefix>/approvals/<id>)`, and a
pending review stage produces a prefixed task link. The old code
produces `/approvals/<id>`.
- `runtime-hiring.test.ts`: unprefixed links that the redirect lands on
the right approval pass. The old grader rejects them. A link that did
not land, landed on another approval, was not clicked, or is another
href still fails.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog`: all passed (one negative control in `internal.test.ts`
changed from a reply wording to "approved while listing").
- `pnpm test:e2e:runner:typecheck` and `server` `tsc --noEmit`: passed.
- `server` vitest for `native-completion-feedback`,
`native-current-wake-completion-feedback` and
`heartbeat/run-completion`: 201 tests passed.
- Narrow paid run of the four full case ids on this branch: run
43657390762 (results are added in a comment).

## Risks

- Low risk. The product change only adds the company prefix to links in
agent feedback text. It adds one indexed company read, and only on the
branches that return a link.
- Less is graded automatically, by design. `skills.author-skill` no
longer checks that the skill keeps the keyboard-only and visible-label
checks or that it does not waive a check. `inbox-search.show-decisions`
no longer checks how the reply names resolvers or separates the person's
decision from others' and from history. `skills.author-skill` no longer
checks whether the reply describes the assignment scope truthfully. A
person reviews this meaning. Saved state and links still catch the
plausible broken outcomes listed above.
- `agents.hire-needs-approval` now needs a reply link that really opens
the approval. A reply without a link to its approval fails as before.

## Model Used

- Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use
and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 05:34:56 -06:00

163 lines
16 KiB
TypeScript

import { chatCatalogNativePhaseDeadline, chatCatalogNativePhaseIsTimely } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import { pollUntil } from "../api.js";
import { conversationLinkLocator } from "../chat-rendered-links.js";
import { sendChatMessage } from "../chat-flow.js";
import { checks, type Entity } from "./internal-support.js";
import { runtimeDefinition, runtimeObservation, observeRuntime, unchangedAgent, type RuntimeAgent, type RuntimeObservation } from "./runtime-support.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
interface HireApproval { id: string; companyId: string; type: string; status: string; requestedByAgentId: string | null;
decidedByUserId: string | null; payload: { agentId?: string; name?: string; role?: string; title?: string; reportsTo?: string; adapterType?: string; adapterConfig?: unknown } }
async function pendingHire(ctx: ChatCatalogContext, name: string) {
return pollUntil({ label: `Actual governed ${name} proposal`, deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 180_000),
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns(),
approvals: await ctx.input.api.get<HireApproval[]>(`/api/companies/${ctx.input.fixtures.company.id}/approvals`),
agents: await ctx.input.api.get<RuntimeAgent[]>(`/api/companies/${ctx.input.fixtures.company.id}/agents`) }; },
accept: value => chatCatalogNativePhaseIsTimely(ctx) && value.source?.conversationState === "waiting" && value.approvals.filter(row => row.type === "hire_agent" && row.status === "pending" && row.payload.name === name).length === 1
&& value.agents.some(row => row.name === name && row.status === "pending_approval")
&& value.runs.every(run => run.agentId === ctx.input.fixtures.agent.id && run.status === "succeeded"),
reject: value => value.runs.length > 4 || value.runs.some(run => run.agentId !== ctx.input.fixtures.agent.id || ["failed", "cancelled", "timed_out"].includes(run.status)) ? "Governed hire admitted unexpected work or failed" : undefined });
}
/** A same-origin board path to one approval: `/<prefix>/approvals/<id>`, or `/approvals/<id>`, which the UI redirects. */
function approvalPath(origin: string, href: string, approvalId: string, prefixed: boolean) {
try {
const url = new URL(href, origin); const parts = url.pathname.replace(/\/+$/, "").split("/").map(decodeURIComponent);
if (url.origin !== new URL(origin).origin || parts[1] === "api") return false;
return prefixed ? parts.length === 4 && !!parts[1] && parts[2] === "approvals" && parts[3] === approvalId
: (parts.length === 3 && parts[1] === "approvals" && parts[2] === approvalId) || (parts.length === 4 && !!parts[1] && parts[2] === "approvals" && parts[3] === approvalId);
} catch { return false; }
}
/** The reply's rendered links that point at this approval's board page, prefixed or through the UI redirect. */
export function replyApprovalLinks(o: Pick<RuntimeObservation, "origin" | "renderedLinkEvidence">, commentIds: unknown, approvalId: string | undefined) {
const ids = Array.isArray(commentIds) ? commentIds.map(String) : [];
return !approvalId || !ids.length ? [] : (o.renderedLinkEvidence ?? []).filter(link => link.source === "reply" && !!link.commentId && ids.includes(link.commentId)
&& approvalPath(o.origin, link.href, approvalId, false));
}
export interface OpenedApprovalLink { href: string | null; approvalId: string; landedPath: string | null; nameVisible: boolean }
/**
* The reply reports the pending approval by giving the person the exact approval to open and decide. Its wording is
* not graded; whether the hire ran early is graded from saved state and runs. The harness clicks the reply's link, so
* an unprefixed link counts only when the UI really lands on that approval's company page showing the proposed hire.
*/
export function repliesLinkApproval(o: RuntimeObservation, commentIds: unknown, approvalId: string | undefined, opened: unknown, companyPrefix: unknown) {
const click = opened as Partial<OpenedApprovalLink> | undefined;
return !!approvalId && typeof companyPrefix === "string" && !!companyPrefix && click?.approvalId === approvalId && click.nameVisible === true
&& typeof click.href === "string" && replyApprovalLinks(o, commentIds, approvalId).some(link => link.href === click.href)
&& typeof click.landedPath === "string" && click.landedPath === `/${encodeURIComponent(companyPrefix)}/approvals/${approvalId}`
&& approvalPath(o.origin, click.landedPath, approvalId, true);
}
export function gradeGovernedHire(value: unknown) {
const c = checks(); const o = runtimeObservation(value);
c.add("independent-governed-hire-evidence", !!o, "Actual pending proposals, human decisions, preserved intended identity and no premature worker execution are required."); if (!o) return c.result();
const rejected = o.state.firstProposal as HireApproval | undefined; const accepted = o.state.secondProposal as HireApproval | undefined;
const approvals = o.state.approvalsAfter as HireApproval[] | undefined;
const first = approvals?.find(row => row.id === rejected?.id); const second = approvals?.find(row => row.id === accepted?.id);
c.add("one-proposal-per-explicit-request", !!rejected && !!accepted && rejected.id !== accepted.id && approvals?.length === 2
&& rejected.companyId === o.companyId && accepted.companyId === o.companyId
&& [rejected, accepted].every(row => row.type === "hire_agent" && row.status === "pending" && row.requestedByAgentId === o.leadId
&& row.payload.role === "qa" && !!row.payload.adapterType && row.payload.adapterConfig != null)
&& rejected.payload.name === "Morgan" && accepted.payload.name === "Jules" && o.state.secondRequestAttributed === true,
"The canonical Morgan proposal and separately changed human Jules request each produce one inspectable QA proposal with real configuration.");
c.add("rejected-hire-stays-inactive", first?.status === "rejected" && !!first.decidedByUserId && o.state.rejectedBeforeChangedRequest === true
&& !o.agents.some(row => row.id === rejected?.payload.agentId && row.lifecycleState === "ready")
&& o.state.noRepeatedProposalBeforeChangedRequest === true,
"The genuine human rejection leaves no working Morgan or repeated proposal before the person explicitly changes their request.");
const before = o.state.acceptedBefore as RuntimeAgent | undefined; const current = o.agents.find(row => row.id === accepted?.payload.agentId);
c.add("accepted-same-intended-hire", second?.status === "approved" && !!second.decidedByUserId && !!before && !!current
&& current.name === "Jules" && current.companyId === o.companyId && current.lifecycleState === "ready"
&& ["active", "idle"].includes(current.status ?? "") && unchangedAgent(before, current),
"Human acceptance activates the exact proposed Jules identity and admitted configuration, rather than a replacement employee.");
c.add("no-worker-before-or-after-decision", o.state.noWorkerBeforeFirstDecision === true && o.state.noWorkerBeforeSecondDecision === true
&& o.runs.length >= 2 && o.runs.length <= 4 && o.runs.every(run => run.agentId === o.leadId && run.status === "succeeded") && o.tasks.length === 0,
"Neither proposed employee executes before approval, and no unrequested task or employee work is created by either decision.");
const extra = o.runs.filter(run => !(o.state.requestRunIds as string[]).includes(run.id));
c.add("bounded-authoritative-decision-notifications", extra.every(run => ["approval.approved", "approval.rejected"].includes(String(run.contextSnapshot?.source))
&& [rejected?.id, accepted?.id].includes(String(run.contextSnapshot?.approvalId))),
"Any extra provider turn is independently tied to one of these exact real human approval decisions.");
c.add("reply-links-each-pending-approval", repliesLinkApproval(o, o.state.firstReplyCommentIds, rejected?.id, o.state.firstApprovalLink, o.state.companyPrefix)
&& repliesLinkApproval(o, o.state.secondReplyCommentIds, accepted?.id, o.state.secondApprovalLink, o.state.companyPrefix),
"Each request's own reply links the exact pending approval, and clicking that link opens the approval's page (directly or through the board redirect).");
return c.result();
}
/**
* Clicks the request reply's rendered link to this approval and records where the UI lands. Without such a link the
* grade fails; the approval page is still opened directly so the later decision phases keep their evidence.
*/
async function openApprovalFromReply(ctx: ChatCatalogContext, state: Record<string, unknown>, runIds: string[], approval: HireApproval, name: string): Promise<OpenedApprovalLink> {
const page = ctx.input.page; const prefix = ctx.input.fixtures.company.issuePrefix;
if (typeof prefix !== "string" || !prefix) throw new Error("Approval link inspection requires the actual company prefix");
const replyIds = async () => (await ctx.input.api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`))
.filter(row => runIds.includes(String(row.createdByRunId)) && row.authorAgentId === ctx.input.fixtures.agent.id).map(row => row.id);
// The reply can settle before its links render in the thread; wait briefly for them.
const found = await pollUntil({ label: `${name} approval link rendered in its reply`, deadlineAt: Date.now() + 20_000,
load: async () => { const o = await observeRuntime(ctx, state); return { o, links: replyApprovalLinks(o, await replyIds(), approval.id) }; },
accept: value => value.links.length > 0 }).catch(() => undefined);
const receipt = found?.links[0];
const landed = (url: URL) => url.pathname === `/${encodeURIComponent(prefix)}/approvals/${approval.id}`;
if (receipt) {
await conversationLinkLocator(page, receipt).first().click();
await page.waitForURL(landed).catch(() => undefined);
}
const landedPath = receipt && landed(new URL(page.url())) ? new URL(page.url()).pathname : null;
if (!landedPath) await page.goto(`/${prefix}/approvals/${approval.id}`, { waitUntil: "domcontentloaded" });
await expect(page.getByText(name, { exact: true }).first()).toBeVisible();
const opened = { href: receipt?.href ?? null, approvalId: approval.id, landedPath, nameVisible: !!landedPath };
await ctx.input.evidence(`chat-catalog-${name.toLowerCase()}-approval-link.json`, { ...opened, url: page.url() });
return opened;
}
export const runtimeHiringCases = [runtimeDefinition("agents.hire-needs-approval", async ctx => {
await ctx.input.api.patch(`/api/companies/${ctx.input.fixtures.company.id}`, { requireBoardApprovalForNewAgents: true });
const company = await ctx.input.api.get<{ requireBoardApprovalForNewAgents: boolean }>(`/api/companies/${ctx.input.fixtures.company.id}`);
if (!company.requireBoardApprovalForNewAgents) throw new Error("Actual company hire approval policy did not persist");
return { companyRequiresApproval: true };
}, gradeGovernedHire, {
sourceFiles: ["chat-catalog-domains/runtime-hiring.ts", "chat-catalog-domains/runtime-support.ts", "chat-scenarios/agents.yaml"],
oracleVersion: "paperclip.chat-catalog.runtime-hiring.v4",
expectedHumanMessageCount: 2, expectedRunCount: 4, minimumRunCount: 2, attemptTimeoutMs: 720_000,
async act(ctx, state) {
const root = `/api/companies/${ctx.input.fixtures.company.id}`; const source = ctx.input.page.url();
const initial = await pendingHire(ctx, "Morgan"); const first = initial.approvals.find(row => row.payload.name === "Morgan" && row.status === "pending")!;
state.firstProposal = first; state.noWorkerBeforeFirstDecision = initial.runs.every(run => run.agentId === ctx.input.fixtures.agent.id);
state.firstReply = (await observeRuntime(ctx, state)).reply; state.companyPrefix = ctx.input.fixtures.company.issuePrefix;
const firstRequestRuns = initial.runs.filter(run => run.contextSnapshot?.source !== "approval.approved" && run.contextSnapshot?.source !== "approval.rejected").map(run => run.id);
state.firstApprovalLink = await openApprovalFromReply(ctx, state, firstRequestRuns, first, "Morgan");
await ctx.input.page.getByRole("button", { name: "Reject", exact: true }).click();
const rejected = await pollUntil({ label: "Rejected Morgan stays inactive without a repeated proposal", deadlineAt: Date.now() + 60_000,
load: async () => ({ approvals: await ctx.input.api.get<HireApproval[]>(`${root}/approvals`), agent: await ctx.input.api.get<RuntimeAgent>(`/api/agents/${first.payload.agentId}`) }),
accept: value => value.approvals.find(row => row.id === first.id)?.status === "rejected" && value.agent.lifecycleState === "rejected" });
state.rejectedBeforeChangedRequest = true; state.noRepeatedProposalBeforeChangedRequest = rejected.approvals.length === 1;
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
const prior = (await ctx.allRuns()).map(run => run.id);
const changed = "I rejected Morgan. I am changing the request: propose a separate QA engineer named Jules using our currently admitted coding account and runtime. Do not create a task or start employee work. Keep the required board approval pending so I can inspect and accept that one intended hire.";
await sendChatMessage(ctx.input.page, changed);
const secondPhase = await pendingHire(ctx, "Jules"); const second = secondPhase.approvals.find(row => row.payload.name === "Jules" && row.status === "pending")!;
state.secondProposal = second; state.noWorkerBeforeSecondDecision = secondPhase.runs.every(run => run.agentId === ctx.input.fixtures.agent.id);
const comments = await ctx.input.api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`);
const human = comments.filter(row => row.body === changed && !row.authorAgentId);
const next = secondPhase.runs.filter(run => !prior.includes(run.id) && !["approval.approved", "approval.rejected"].includes(String(run.contextSnapshot?.source)));
const profiles = await Promise.all(next.map(run => ctx.input.api.get<{ runnerProfileJson?: { nativeExecutionInput?: { task?: { prompt?: string } } } }>(`/api/heartbeat-runs/${run.id}`)));
state.secondRequestAttributed = human.length === 1 && next.length === 1 && profiles.some(run => run.runnerProfileJson?.nativeExecutionInput?.task?.prompt?.includes(human[0]!.id));
state.requestRunIds = [...firstRequestRuns, ...next.map(run => run.id)];
const leadReplies = (runIds: string[]) => comments.filter(row => runIds.includes(String(row.createdByRunId)) && row.authorAgentId === ctx.input.fixtures.agent.id);
state.firstReplyCommentIds = leadReplies(firstRequestRuns).map(row => row.id);
state.secondReplyCommentIds = leadReplies(next.map(run => run.id)).map(row => row.id);
state.secondReply = leadReplies(next.map(run => run.id)).map(row => row.body ?? "").join("\n");
state.acceptedBefore = secondPhase.agents.find(row => row.id === second.payload.agentId);
state.worker = state.acceptedBefore;
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
state.secondApprovalLink = await openApprovalFromReply(ctx, state, next.map(run => run.id), second, "Jules");
await ctx.input.page.getByRole("button", { name: "Approve", exact: true }).click();
await pollUntil({ label: "Accepted exact QA hire becomes genuinely admitted", deadlineAt: Date.now() + 90_000,
load: () => ctx.input.api.get<RuntimeAgent>(`/api/agents/${second.payload.agentId}`),
accept: row => row.lifecycleState === "ready" && ["idle", "active"].includes(row.status ?? ""),
reject: row => row.lifecycleState === "error" ? "Approved runtime did not qualify; no fake ready state is permitted" : undefined });
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
await expect(ctx.input.page.getByText("approved", { exact: true }).first()).toBeVisible();
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
},
async observe(ctx, state) {
state.approvalsAfter = await ctx.input.api.get<HireApproval[]>(`/api/companies/${ctx.input.fixtures.company.id}/approvals`);
return observeRuntime(ctx, state);
},
})];