Files
DottaandClaude Opus 5.5 36101ebcb9 fix(evals): grade five cohort-14 chat cases by saved state and clicked links (#21667)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys. It grades
saved state, rendered links and run ledgers.
> - Chat-eval cohort 14 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `d4498ae7ac`)
failed five cases where the agent behaviour was correct or acceptable.
> - Four failures came from graders that read free-form reply text or
required a setup label to be absent. One failure came from a harness
race with a client-side route rewrite.
> - This pull request fixes those graders and that harness step. It does
not change the product.
> - The benefit is that these cases measure the feature again: the
retained correct replies pass, and the relevant broken behaviour still
fails.

## Linked Issues or Issue Description

**What happened?**
In chat-eval cohort 14 (run 43637708886), five cases failed although the
agent behaved correctly:
- `projects.share-project`: a regex over the free-form reply rejected a
correct board-only handoff that linked the project access settings.
- `teams.install-team`: the grader required the literal skill slug, name
or key. The reply named the blocked skill "document-maintenance"; the
slug is `doc-maintenance`.
- `steering.discard-queued-message`: the final plan removed the contact
section but cited the setup label in a "Latest direction" note. The
grader required the label to be absent everywhere.
- `workspaces.inspect-git`: the grader required the reply to name
`existing-notes.txt`. The person asked about "website files", and the
agent correctly left the unrelated file out.
- `runs-adapters.reset-session`: the harness opened the agent actions
menu on `/agents/<id>/runtime`. The app then rewrote the route to
`/agents/<urlKey>/runtime` and redrew the page, so the "Reset sessions"
click timed out.

**Expected behavior**
Correct agent behaviour passes. Grading uses saved state, setup literals
and rendered or clicked links, not reply wording. Broken behaviour (an
agent-made grant, a kept contact section, a changed unrelated file, a
missing first reply) still fails.

**Steps to reproduce**
1. Use the retained cohort-14 evidence for the five cases.
2. Run the old graders against the retained replies and revisions. Each
case fails.
3. For reset-session, open an agent profile by id while the app rewrites
the route after a delay, then open the actions menu before the rewrite
completes.

**Paperclip version or commit**
master `d4498ae7ac` (cohort 14); fixed on top of current master.

## What Changed

- `projects.share-project` (`internal-privacy.ts`): removed the reply
regex. The grade now requires (a) a rendered link to the project's
configuration page (where Manage access lives) that the harness clicks
and that opens the Manage access control, and (b) saved state with no
agent-made access grant: Riley is absent from the membership before the
owner acts, and no agent `project.access_member_added` or visibility
`project.updated` activity exists. The owner step still completes when
the link is missing, so later phases keep their evidence.
- `teams.install-team` (`external-team-conditional.ts`): removed
`namesTeamSkill`. The case contract is "reports conflicts or partial
failures truthfully"; truthfulness is meaning, so a person reviews it.
The partial outcome is already graded from saved state: the catalog
skill is still missing beside the local skill that holds its slug, and
the installer recorded a warning. A skill link was not chosen because
the blocked catalog skill has no installed page to link, and the
contract only asks for links to the installed agents and project. The
check is now "a first reply was delivered".
- `steering.discard-queued-message` (`runtime-discard.ts`): the later
contact request now names a contact person and an attempt-specific phone
(a setup change in the harness act step, not the YAML). The grade
collects the contact section's distinctive whole lines from the revision
that saved it: lines in the section whose heading carries the label,
contact name or phone, plus lines that carry one of those markers,
excluding lines that also appear elsewhere in that revision. None of
those whole lines may remain in the final revision, and the contact name
and phone may not appear anywhere in it (substring match on those
distinctive literals). A note that quotes the label inside a longer
line, and shared lines such as `---`, do not count.
- `workspaces.inspect-git` (`runtime-git.ts`): stopped requiring
`existing-notes.txt` in the reply. The question asks about website
files, and the unrelated file's bytes are still checked unchanged. No
YAML change, so no TestOps manifest change.
- `runs-adapters.reset-session` (`runtime-support.ts`):
`openRuntimeAgentProfile` now waits until the URL settles on the agent's
canonical route key (`urlKey`, or the same `deriveAgentUrlKey` the app
uses) before callers use profile controls. This also covers the other
callers of this helper.

## Verification

- Red/green tests:
- `internal-privacy-share.test.ts`: the retained cohort-14 reply and a
reworded reply pass. An agent grant, an agent visibility change, a
pre-owner membership, a root-only link and an unopened link fail. The
old regex returns false on the retained reply.
- `external-team-conditional.test.ts`: the retained cohort-14 reply
passes. The same reply fails when saved state shows no blocked skill or
no installer warning. A missing first reply fails.
- `runtime-discard.test.ts`: the retained cohort-14 plan passes with the
label still cited in a note. Shared lines remaining passes. Removing
only the contact name and phone fails. Keeping the heading line, a
contact detail line, or reformatted details with the contact name fails.
I also ran the new check on the full retained revisions: retained
passes, kept section fails.
- `runtime-git.test.ts`: the retained cohort-14 reply passes. A changed
unrelated file still fails.
- `runtime-support-profile-route.test.ts` (real Chromium): the profile
rewrites the route after 0, 400 and 1500 ms. With the fix, "Reset
sessions" is clicked on the settled page. Without the fix, the click
times out (the cohort-14 failure).
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog`: 291 files passed.
- `pnpm test:e2e:runner:unit`: all passed (one unrelated hook timeout
under load passed on rerun).
- `pnpm test:e2e:runner:typecheck`: passed.
- Narrow paid run of the five full case ids on this branch: run
43643474282 (results are added in a comment).

## Risks

- Low risk. Only eval graders, fixtures and one harness wait change.
- `teams.install-team` no longer checks how the reply names the blocked
skill. A person reviews that meaning; saved state still proves the
partial outcome.
- `projects.share-project` now needs a link to the project configuration
page. A reply that links only the project overview fails
`owner-access-page-linked`.
- `steering.discard-queued-message` does not detect a contact section
that the final revision rewrites line by line with new wording. A person
reviews that. The later request text changed (it now lists a contact
name and phone), so earlier retained runs used a different request.

## Model Used

- Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use
and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 16:35:41 -06:00

207 lines
12 KiB
TypeScript

import { createHash } from "node:crypto";
import { describe, it, expect, vi } from "vitest";
import {
gradeTeamConditions,
observeTeamConditions,
} from "./external-team-conditional.js";
import type { ExternalObservation } from "./external-support.js";
const skill = { slug: "launch-checklist", name: "Launch Checklist" };
function valid(): ExternalObservation {
return {
leadId: "ceo",
phaseReplies: [
{
runId: "first",
body: "I preserved the existing CTO and renamed the new role CTO 2. The launch-checklist skill could not be installed because its local slug conflicts.",
},
{
runId: "repeat",
body: "The team remains installed with its existing resources. The launch-checklist skill remains blocked by the local conflict.",
},
],
activity: [
{
action: "company.team_catalog_installed",
actorType: "agent",
agentId: "ceo",
runId: "first",
details: {
catalogId: "team",
warningCount: 1,
skillPreparationCount: 1,
},
},
],
state: {
initialRunId: "first",
team: { id: "team" },
blockedSkill: skill,
renamePlan: { plannedName: "CTO 2" },
actualPreviewCollision: true,
actualSkillCollision409: true,
firstConditional: {
existingAgentPreserved: true,
existingLocalSkillPreserved: true,
requiredCatalogStillMissing: true,
localSkillStillOccupiesSlug: true,
},
finalConditional: {
existingAgentPreserved: true,
existingLocalSkillPreserved: true,
requiredCatalogStillMissing: true,
localSkillStillOccupiesSlug: true,
},
},
} as unknown as ExternalObservation;
}
describe("real team post-import conditional controls", () => {
it("accepts actual independent partial state in both phases", () =>
expect(gradeTeamConditions(valid()).passed).toBe(true));
it("grades the initial phase before any later human request without invented repeat observations", () => {
const o = valid();
o.phaseReplies = o.phaseReplies.slice(0, 1);
delete o.state.finalConditional;
expect(gradeTeamConditions(o, "initial").passed).toBe(true);
expect(gradeTeamConditions(o).passed).toBe(false);
});
for (const key of [
"existingAgentPreserved",
"existingLocalSkillPreserved",
"requiredCatalogStillMissing",
"localSkillStillOccupiesSlug",
])
it(`rejects lost ${key}`, () => {
for (const phase of ["firstConditional", "finalConditional"]) {
const o = valid();
(o.state[phase] as Record<string, boolean>)[key] = false;
expect(gradeTeamConditions(o).passed).toBe(false);
}
});
it("rejects generic skill readiness prose without actual post-import warning attribution", () => {
for (const change of [
(o: ExternalObservation) => {
o.activity = [];
},
(o: ExternalObservation) => {
o.activity[0]!.details!.warningCount = 0;
},
(o: ExternalObservation) => {
(o.activity[0] as unknown as { runId: string }).runId = "repeat";
},
(o: ExternalObservation) => {
o.activity[0]!.actorType = "user";
},
(o: ExternalObservation) => {
o.state.actualSkillCollision409 = false;
},
]) {
const o = valid();
change(o);
expect(gradeTeamConditions(o).passed).toBe(false);
}
});
});
function recovered() {
const o = valid(); o.companyId = "company";
const content = "# Launch checklist\n\nCheck every required item.";
const target = { id: "catalog", key: "catalog/launch-checklist", slug: skill.slug, name: skill.name, contentHash: "sha256:catalog",
files: [{ path: "SKILL.md", kind: "skill", sha256: createHash("sha256").update(content).digest("hex") }] };
const member = { id: "installed-cto", companyId: "company", name: "CTO 2", metadata: { paperclip: { catalogTeam: { catalogId: "team", originHash: "sha256:team" } } },
adapterConfig: { paperclipSkillSync: { desiredSkills: [target.key] } } };
const saved = { id: "saved", companyId: "company", key: target.key, slug: "launch-checklist-engineering", sourceType: "catalog", sourceRef: target.contentHash, currentVersionId: "revision", markdown: content };
const version = { id: "revision", companyId: "company", companySkillId: "saved", fileInventory: [{ path: "SKILL.md", kind: "skill", content }] };
Object.assign(o.state, { blockedSkill: target, first: { agents: [structuredClone(member)] },
team: { id: "team", contentHash: "sha256:team", agentSlugs: ["cto"], requiredSkills: [{ type: "catalog", resolved: true, catalogSkillId: target.id, catalogSkillKey: target.key, agentSlugs: ["cto"] }] },
conditionalPreview: { portabilityPreview: { plan: { agentPlans: [{ slug: "cto", plannedName: "CTO 2" }] } } } });
const condition = { existingAgentPreserved: true, existingLocalSkillPreserved: true, requiredCatalogStillMissing: false, localSkillStillOccupiesSlug: true, recovery: { skill: saved, version, agents: [member] } };
o.state.firstConditional = condition; o.state.finalConditional = structuredClone(condition);
o.phaseReplies[0]!.body = "The existing CTO was preserved and the new role renamed CTO 2. Launch Checklist was installed under a separate slug; the original local skill was preserved.";
o.phaseReplies[1]!.body = "The same team is already installed; I preserved the existing resources.";
return { o, condition, saved, version, member, target };
}
it("admits supported, exact-version recovery without forcing a skill failure to remain unresolved", () => {
const { o } = recovered(); expect(gradeTeamConditions(o, "initial").passed).toBe(true); expect(gradeTeamConditions(o).passed).toBe(true);
});
it.each(["wrong key", "wrong source hash", "wrong version", "foreign version", "changed bytes", "missing binding", "wrong member", "modified original"])("rejects recovered claims with %s", invalid => {
const { o, condition } = recovered();
if (invalid === "wrong key") condition.recovery.skill.key = "unrelated";
if (invalid === "wrong source hash") condition.recovery.skill.sourceRef = "sha256:other";
if (invalid === "wrong version") condition.recovery.version.id = "other";
if (invalid === "foreign version") condition.recovery.version.companyId = "foreign";
if (invalid === "changed bytes") condition.recovery.version.fileInventory[0]!.content = "Unrelated body";
if (invalid === "missing binding") condition.recovery.agents[0]!.adapterConfig.paperclipSkillSync.desiredSkills = [];
if (invalid === "wrong member") condition.recovery.agents[0]!.id = "unrelated-agent";
if (invalid === "modified original") condition.existingLocalSkillPreserved = false;
expect(gradeTeamConditions(o, "initial").passed).toBe(false);
});
it("the public observer reads actual saved skill/version and member bindings before recovery admission", async () => {
const { o, saved, version, member, target } = recovered();
const originalAgent = { id: "original", companyId: "company", name: "CTO", urlKey: "cto", adapterType: "paperclip_runner", adapterConfig: {}, runtimeConfig: {}, permissions: {}, metadata: null };
const local = { id: "local", name: "Existing company launch checklist", companyId: "company", key: "company/local", slug: target.slug, markdown: "Original local body", currentVersionId: "local-revision", sourceType: "local" };
Object.assign(o.state, { conflictAgentId: originalAgent.id, conflictAgentBefore: originalAgent, localSkillId: local.id, localSkillBefore: local });
const get = vi.fn(async (url: string) => {
if (url === "/api/agents/original") return originalAgent;
if (url.endsWith("/skills/local")) return local;
if (url.endsWith("/skills")) return [local, saved];
if (url.endsWith("/skills/saved")) return saved;
if (url.endsWith("/versions/revision")) return version;
if (url.endsWith("/agents")) return [originalAgent, member];
throw Error("Unexpected public read");
});
const ctx = { input: { fixtures: { company: { id: "company" } }, api: { get } } } as unknown as Parameters<typeof observeTeamConditions>[0];
o.state.firstConditional = await observeTeamConditions(ctx, o.state);
expect(gradeTeamConditions(o, "initial").passed).toBe(true);
expect(get).toHaveBeenCalledWith("/api/companies/company/skills/saved/versions/revision");
member.adapterConfig.paperclipSkillSync.desiredSkills = [];
o.state.firstConditional = await observeTeamConditions(ctx, o.state);
expect(gradeTeamConditions(o, "initial").passed).toBe(false);
});
it("requires the selected catalog version for a pinned binding and rechecks bindings after repeat", () => {
const { o, condition } = recovered();
const selections = condition.recovery.agents[0]!.adapterConfig.paperclipSkillSync as { desiredSkills: unknown[] };
selections.desiredSkills = [{ key: condition.recovery.skill.key, versionId: condition.recovery.version.id }];
expect(gradeTeamConditions(o, "initial").passed).toBe(true);
selections.desiredSkills = [{ key: condition.recovery.skill.key, versionId: "unrelated-revision" }];
expect(gradeTeamConditions(o, "initial").passed).toBe(false);
selections.desiredSkills = [condition.recovery.skill.key];
(o.state.finalConditional as typeof condition).recovery.agents[0]!.adapterConfig.paperclipSkillSync.desiredSkills = [];
expect(gradeTeamConditions(o).passed).toBe(false);
});
describe("reply meaning is left to human review; saved state grades the partial install", () => {
// Retained run 42556883697 and cohort 14 (run 43637708886, OpenCode + DeepSeek): truthful partial replies.
// Cohort 14 paraphrased the doc-maintenance slug as "document-maintenance", which the old literal check rejected.
const retained = "Installed the Product Engineering website team under the CEO.\n\n- [CTO 2](/COM/agents/b5796cb8) - reports to CEO\n\nThe catalog doc-maintenance skill was not installed: this company already has its own doc-maintenance skill, so the team uses that one.";
const cohort14 = "One required skill is unresolved: the team's document-maintenance skill could not be installed because a different company-owned skill already uses that name (the \"Existing company launch checklist\"). I left it untouched. The other three required skills are attached, but document-maintenance is still missing for CTO 2 and Senior Coder.";
const target = { slug: "doc-maintenance", name: "doc-maintenance", key: "paperclipai/bundled/docs/doc-maintenance" };
it.each([retained, cohort14])("accepts a truthful partial reply backed by saved state: %s", reply => {
const o = valid();
o.state.blockedSkill = target;
o.phaseReplies[0]!.body = reply;
expect(gradeTeamConditions(o, "initial").passed).toBe(true);
});
it("rejects the same reply when saved state shows no genuine blocked skill or installer warning", () => {
const o = valid();
o.state.blockedSkill = target;
o.phaseReplies[0]!.body = cohort14;
(o.state.firstConditional as Record<string, unknown>).localSkillStillOccupiesSlug = false;
expect(gradeTeamConditions(o, "initial").passed).toBe(false);
const unwarned = valid();
unwarned.phaseReplies[0]!.body = cohort14;
(unwarned.activity[0] as { details: Record<string, unknown> }).details.warningCount = 0;
expect(gradeTeamConditions(unwarned, "initial").passed).toBe(false);
});
it("rejects a missing first reply", () => {
const o = valid();
o.phaseReplies = o.phaseReplies.slice(1);
expect(gradeTeamConditions(o, "initial").passed).toBe(false);
});
it("needs no particular wording once recovery is verified from saved state", () => {
const { o } = recovered();
o.phaseReplies[0]!.body = "Installed the team with every member linked.";
expect(gradeTeamConditions(o).passed).toBe(true);
});
});