mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 20:29:18 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 14 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `d4498ae7ac`) failed five cases where the agent behaviour was correct or acceptable. > - Four failures came from graders that read free-form reply text or required a setup label to be absent. One failure came from a harness race with a client-side route rewrite. > - This pull request fixes those graders and that harness step. It does not change the product. > - The benefit is that these cases measure the feature again: the retained correct replies pass, and the relevant broken behaviour still fails. ## Linked Issues or Issue Description **What happened?** In chat-eval cohort 14 (run 43637708886), five cases failed although the agent behaved correctly: - `projects.share-project`: a regex over the free-form reply rejected a correct board-only handoff that linked the project access settings. - `teams.install-team`: the grader required the literal skill slug, name or key. The reply named the blocked skill "document-maintenance"; the slug is `doc-maintenance`. - `steering.discard-queued-message`: the final plan removed the contact section but cited the setup label in a "Latest direction" note. The grader required the label to be absent everywhere. - `workspaces.inspect-git`: the grader required the reply to name `existing-notes.txt`. The person asked about "website files", and the agent correctly left the unrelated file out. - `runs-adapters.reset-session`: the harness opened the agent actions menu on `/agents/<id>/runtime`. The app then rewrote the route to `/agents/<urlKey>/runtime` and redrew the page, so the "Reset sessions" click timed out. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not reply wording. Broken behaviour (an agent-made grant, a kept contact section, a changed unrelated file, a missing first reply) still fails. **Steps to reproduce** 1. Use the retained cohort-14 evidence for the five cases. 2. Run the old graders against the retained replies and revisions. Each case fails. 3. For reset-session, open an agent profile by id while the app rewrites the route after a delay, then open the actions menu before the rewrite completes. **Paperclip version or commit** master `d4498ae7ac` (cohort 14); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`): removed the reply regex. The grade now requires (a) a rendered link to the project's configuration page (where Manage access lives) that the harness clicks and that opens the Manage access control, and (b) saved state with no agent-made access grant: Riley is absent from the membership before the owner acts, and no agent `project.access_member_added` or visibility `project.updated` activity exists. The owner step still completes when the link is missing, so later phases keep their evidence. - `teams.install-team` (`external-team-conditional.ts`): removed `namesTeamSkill`. The case contract is "reports conflicts or partial failures truthfully"; truthfulness is meaning, so a person reviews it. The partial outcome is already graded from saved state: the catalog skill is still missing beside the local skill that holds its slug, and the installer recorded a warning. A skill link was not chosen because the blocked catalog skill has no installed page to link, and the contract only asks for links to the installed agents and project. The check is now "a first reply was delivered". - `steering.discard-queued-message` (`runtime-discard.ts`): the later contact request now names a contact person and an attempt-specific phone (a setup change in the harness act step, not the YAML). The grade collects the contact section's distinctive whole lines from the revision that saved it: lines in the section whose heading carries the label, contact name or phone, plus lines that carry one of those markers, excluding lines that also appear elsewhere in that revision. None of those whole lines may remain in the final revision, and the contact name and phone may not appear anywhere in it (substring match on those distinctive literals). A note that quotes the label inside a longer line, and shared lines such as `---`, do not count. - `workspaces.inspect-git` (`runtime-git.ts`): stopped requiring `existing-notes.txt` in the reply. The question asks about website files, and the unrelated file's bytes are still checked unchanged. No YAML change, so no TestOps manifest change. - `runs-adapters.reset-session` (`runtime-support.ts`): `openRuntimeAgentProfile` now waits until the URL settles on the agent's canonical route key (`urlKey`, or the same `deriveAgentUrlKey` the app uses) before callers use profile controls. This also covers the other callers of this helper. ## Verification - Red/green tests: - `internal-privacy-share.test.ts`: the retained cohort-14 reply and a reworded reply pass. An agent grant, an agent visibility change, a pre-owner membership, a root-only link and an unopened link fail. The old regex returns false on the retained reply. - `external-team-conditional.test.ts`: the retained cohort-14 reply passes. The same reply fails when saved state shows no blocked skill or no installer warning. A missing first reply fails. - `runtime-discard.test.ts`: the retained cohort-14 plan passes with the label still cited in a note. Shared lines remaining passes. Removing only the contact name and phone fails. Keeping the heading line, a contact detail line, or reformatted details with the contact name fails. I also ran the new check on the full retained revisions: retained passes, kept section fails. - `runtime-git.test.ts`: the retained cohort-14 reply passes. A changed unrelated file still fails. - `runtime-support-profile-route.test.ts` (real Chromium): the profile rewrites the route after 0, 400 and 1500 ms. With the fix, "Reset sessions" is clicked on the settled page. Without the fix, the click times out (the cohort-14 failure). - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: 291 files passed. - `pnpm test:e2e:runner:unit`: all passed (one unrelated hook timeout under load passed on rerun). - `pnpm test:e2e:runner:typecheck`: passed. - Narrow paid run of the five full case ids on this branch: run 43643474282 (results are added in a comment). ## Risks - Low risk. Only eval graders, fixtures and one harness wait change. - `teams.install-team` no longer checks how the reply names the blocked skill. A person reviews that meaning; saved state still proves the partial outcome. - `projects.share-project` now needs a link to the project configuration page. A reply that links only the project overview fails `owner-access-page-linked`. - `steering.discard-queued-message` does not detect a contact section that the final revision rewrites line by line with new wording. A person reviews that. The later request text changed (it now lists a contact name and phone), so earlier retained runs used a different request. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
279 lines
13 KiB
TypeScript
279 lines
13 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { isDeepStrictEqual } from "node:util";
|
|
import type { Agent } from "../../../packages/shared/src/types/agent.js";
|
|
import type {
|
|
CatalogSkill,
|
|
CompanySkill,
|
|
CompanySkillListItem,
|
|
CompanySkillDetail,
|
|
CompanySkillVersion,
|
|
} from "../../../packages/shared/src/types/company-skill.js";
|
|
import type {
|
|
CatalogTeam,
|
|
CatalogTeamImportPreviewResult,
|
|
} from "../../../packages/shared/src/types/teams-catalog.js";
|
|
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
|
|
import { RunnerApiHttpError } from "../api.js";
|
|
import { identityWorker } from "./identity-support.js";
|
|
import { checks, root, type ExternalObservation } from "./external-support.js";
|
|
import type { Entity } from "./internal-support.js";
|
|
const agentAuthority = (a: Agent) => ({
|
|
id: a.id,
|
|
companyId: a.companyId,
|
|
name: a.name,
|
|
urlKey: a.urlKey,
|
|
adapterType: a.adapterType,
|
|
adapterConfig: a.adapterConfig,
|
|
runtimeConfig: a.runtimeConfig,
|
|
permissions: a.permissions,
|
|
metadata: a.metadata,
|
|
});
|
|
const skillIdentity = (s: CompanySkill) => ({
|
|
id: s.id,
|
|
name: s.name,
|
|
companyId: s.companyId,
|
|
key: s.key,
|
|
slug: s.slug,
|
|
markdown: s.markdown,
|
|
currentVersionId: s.currentVersionId,
|
|
sourceType: s.sourceType,
|
|
});
|
|
/** Genuine post-import failure: the catalog installer imports the team graph,
|
|
* then installFromCatalog rejects this separately owned local skill's occupied
|
|
* slug. No test flag, permission denial, missing account or catalog mutation. */
|
|
export async function prepareTeamConditions(
|
|
ctx: ChatCatalogContext,
|
|
team: CatalogTeam,
|
|
) {
|
|
const preview = await ctx.input.api.post<CatalogTeamImportPreviewResult>(
|
|
`${root(ctx)}/teams/catalog/${encodeURIComponent(team.id)}/preview`,
|
|
{ targetManagerAgentId: ctx.input.fixtures.agent.id },
|
|
);
|
|
const rootAgent = preview.portabilityPreview.manifest.agents.find(
|
|
(a) => a.slug === team.rootAgentSlugs[0],
|
|
);
|
|
const skills = await ctx.input.api.get<CatalogSkill[]>("/api/skills/catalog");
|
|
const required = skills.find((s) =>
|
|
team.requiredSkills.some(
|
|
(r) => r.type === "catalog" && r.resolved && r.catalogSkillId === s.id,
|
|
),
|
|
);
|
|
if (!rootAgent || !required)
|
|
throw new Error(
|
|
"Setup blocker: actual team lacks its inspectable root role/required catalog skill for finite installer conditions",
|
|
);
|
|
const inventory = await ctx.input.api.get<CompanySkillListItem[]>(
|
|
`${root(ctx)}/skills`,
|
|
);
|
|
const occupied = inventory.filter((s) => s.slug === required.slug);
|
|
// A default installed catalog skill in this fresh owned company can be
|
|
// replaced with a real local customization through ordinary public controls.
|
|
if (occupied.some((s) => s.key !== required.key))
|
|
throw new Error(
|
|
"Setup blocker: unexpected preexisting local skill already occupies selected conditional fixture",
|
|
);
|
|
for (const s of occupied)
|
|
await ctx.input.api.delete(`${root(ctx)}/skills/${s.id}`);
|
|
const local = await ctx.input.api.post<CompanySkill>(`${root(ctx)}/skills`, {
|
|
name: "Existing company launch checklist",
|
|
slug: required.slug,
|
|
description: "Company-owned local launch review checklist",
|
|
markdown:
|
|
"---\nname: Existing company launch checklist\ndescription: Company-owned local launch review checklist\n---\n\nCheck the launch owner and release checklist before publishing.\n",
|
|
});
|
|
if (local.key === required.key || local.slug !== required.slug)
|
|
throw new Error(
|
|
"Real local skill did not create the intended separate-key slug conflict",
|
|
);
|
|
let denied409 = false;
|
|
try {
|
|
await ctx.input.api.post(`${root(ctx)}/skills/install-catalog`, {
|
|
catalogSkillId: required.id,
|
|
});
|
|
} catch (error) {
|
|
if (error instanceof RunnerApiHttpError && error.status === 409)
|
|
denied409 = true;
|
|
else throw error;
|
|
}
|
|
if (!denied409)
|
|
throw new Error(
|
|
"Setup blocker: actual required-skill installer did not reject the real occupied local slug",
|
|
);
|
|
const created = await identityWorker(ctx, rootAgent.name);
|
|
const existing = await ctx.input.api.get<Agent>(`/api/agents/${created.id}`);
|
|
const conflict = await ctx.input.api.post<CatalogTeamImportPreviewResult>(
|
|
`${root(ctx)}/teams/catalog/${encodeURIComponent(team.id)}/preview`,
|
|
{ targetManagerAgentId: ctx.input.fixtures.agent.id },
|
|
);
|
|
const plan = conflict.portabilityPreview.plan.agentPlans.find(
|
|
(p) => p.slug === rootAgent.slug,
|
|
);
|
|
if (
|
|
!plan ||
|
|
plan.existingAgentId !== existing.id ||
|
|
plan.action !== "create" ||
|
|
plan.plannedName === existing.name ||
|
|
!/rename/i.test(plan.reason ?? "")
|
|
)
|
|
throw new Error(
|
|
"Setup blocker: actual package preview did not prove the unrelated agent-name collision and rename plan",
|
|
);
|
|
return {
|
|
conflictAgentId: existing.id,
|
|
conflictAgentBefore: agentAuthority(existing),
|
|
blockedSkill: required,
|
|
localSkillId: local.id,
|
|
localSkillBefore: skillIdentity(local),
|
|
actualSkillCollision409: true,
|
|
actualPreviewCollision: true,
|
|
renamePlan: plan,
|
|
conditionalPreview: conflict,
|
|
};
|
|
}
|
|
interface TeamSkillRecovery {
|
|
skill: CompanySkillDetail;
|
|
version: CompanySkillVersion;
|
|
agents: Agent[];
|
|
}
|
|
export function verifiedTeamSkillRecovery(state: Record<string, unknown>, companyId: string, value: unknown): boolean {
|
|
const recovery = value as TeamSkillRecovery | undefined, target = state.blockedSkill as CatalogSkill;
|
|
const team = state.team as CatalogTeam, preview = state.conditionalPreview as CatalogTeamImportPreviewResult;
|
|
const first = state.first as { agents: Agent[] } | undefined;
|
|
if (!recovery?.skill || !recovery.version || !Array.isArray(recovery.agents) || !target?.key || !target.contentHash
|
|
|| !team?.id || !preview?.portabilityPreview?.plan || !first?.agents) return false;
|
|
const { skill, version, agents } = recovery;
|
|
if (!skill.id || skill.companyId !== companyId || skill.key !== target.key || skill.sourceType !== "catalog" || skill.sourceRef !== target.contentHash
|
|
|| !skill.currentVersionId || version.id !== skill.currentVersionId || version.companyId !== companyId || version.companySkillId !== skill.id
|
|
|| !Array.isArray(version.fileInventory) || !target.files?.length || version.fileInventory.length !== target.files.length
|
|
|| new Set(version.fileInventory.map(file => file.path)).size !== target.files.length
|
|
|| version.fileInventory.find(file => file.path === "SKILL.md")?.content !== skill.markdown
|
|
|| !target.files.every(expected => {
|
|
const file = version.fileInventory.find(file => file.path === expected.path);
|
|
return file && file.kind === expected.kind && typeof file.content === "string"
|
|
&& createHash("sha256").update(Buffer.from(file.content, file.encoding === "base64" ? "base64" : "utf8")).digest("hex") === expected.sha256;
|
|
})) return false;
|
|
const required = team.requiredSkills.find(row => row.type === "catalog" && row.resolved && row.catalogSkillId === target.id && row.catalogSkillKey === target.key);
|
|
if (!required?.agentSlugs.length) return false;
|
|
const plans = preview.portabilityPreview.plan.agentPlans;
|
|
const members = (rows: Agent[]) => rows.filter(agent => {
|
|
const provenance = (agent.metadata as { paperclip?: { catalogTeam?: { catalogId?: string; originHash?: string } } } | null)?.paperclip?.catalogTeam;
|
|
return agent.companyId === companyId && provenance?.catalogId === team.id && provenance.originHash === team.contentHash;
|
|
});
|
|
const initialMembers = members(first.agents), currentMembers = members(agents);
|
|
if (initialMembers.length !== team.agentSlugs.length || currentMembers.length !== initialMembers.length
|
|
|| new Set(currentMembers.map(agent => agent.id)).size !== currentMembers.length) return false;
|
|
return required.agentSlugs.every(slug => {
|
|
const planned = plans.filter(plan => plan.slug === slug);
|
|
if (planned.length !== 1) return false;
|
|
const initial = initialMembers.filter(agent => agent.name === planned[0]!.plannedName);
|
|
if (initial.length !== 1) return false;
|
|
const current = currentMembers.find(agent => agent.id === initial[0]!.id);
|
|
const sync = current?.adapterConfig?.paperclipSkillSync as { desiredSkills?: unknown[] } | undefined;
|
|
const selection = sync?.desiredSkills?.find(value => typeof value === "string" ? value.trim() === target.key
|
|
: !!value && typeof value === "object" && "key" in value && value.key === target.key);
|
|
return typeof selection === "string" || !!selection && typeof selection === "object" && "versionId" in selection
|
|
&& (selection.versionId === null || selection.versionId === version.id);
|
|
});
|
|
}
|
|
export async function observeTeamConditions(
|
|
ctx: ChatCatalogContext,
|
|
state: Record<string, unknown>,
|
|
) {
|
|
const [agent, skill, skills] = await Promise.all([
|
|
ctx.input.api.get<Agent>(`/api/agents/${state.conflictAgentId}`),
|
|
ctx.input.api.get<CompanySkill>(
|
|
`${root(ctx)}/skills/${state.localSkillId}`,
|
|
),
|
|
ctx.input.api.get<CompanySkillListItem[]>(`${root(ctx)}/skills`),
|
|
]);
|
|
const target = state.blockedSkill as CatalogSkill;
|
|
const installed = skills.filter(row => row.key === target.key && row.sourceType === "catalog");
|
|
let recovery: TeamSkillRecovery | undefined;
|
|
if (installed.length === 1) {
|
|
const detail = await ctx.input.api.get<CompanySkillDetail>(`${root(ctx)}/skills/${encodeURIComponent(installed[0]!.id)}`);
|
|
if (detail.id === installed[0]!.id && detail.currentVersionId) {
|
|
const [version, agents] = await Promise.all([
|
|
ctx.input.api.get<CompanySkillVersion>(`${root(ctx)}/skills/${encodeURIComponent(detail.id)}/versions/${encodeURIComponent(detail.currentVersionId)}`),
|
|
ctx.input.api.get<Agent[]>(`${root(ctx)}/agents`),
|
|
]);
|
|
recovery = { skill: detail, version, agents };
|
|
}
|
|
}
|
|
return {
|
|
existingAgentPreserved: isDeepStrictEqual(
|
|
agentAuthority(agent),
|
|
state.conflictAgentBefore,
|
|
),
|
|
existingLocalSkillPreserved: isDeepStrictEqual(
|
|
skillIdentity(skill),
|
|
state.localSkillBefore,
|
|
),
|
|
recovery,
|
|
requiredCatalogStillMissing: !skills.some(
|
|
(s) => s.key === target.key && s.sourceType === "catalog",
|
|
),
|
|
localSkillStillOccupiesSlug:
|
|
skill.slug === target.slug && skill.key !== target.key,
|
|
};
|
|
}
|
|
export function gradeTeamConditions(
|
|
o: ExternalObservation,
|
|
phase: "initial" | "complete" = "complete",
|
|
) {
|
|
const c = checks(),
|
|
target = o.state.blockedSkill as CatalogSkill | undefined,
|
|
plan = o.state.renamePlan as { plannedName: string } | undefined;
|
|
const first = o.phaseReplies
|
|
.filter((r) => r.runId === o.state.initialRunId)
|
|
.map((r) => r.body)
|
|
.join("\n");
|
|
const earlier = o.state.firstConditional as
|
|
Record<string, unknown> | undefined,
|
|
later = o.state.finalConditional as Record<string, unknown> | undefined;
|
|
c.add(
|
|
"real-preview-conflict-and-owned-originals",
|
|
o.state.actualPreviewCollision === true &&
|
|
o.state.actualSkillCollision409 === true &&
|
|
!!plan &&
|
|
!!target &&
|
|
earlier?.existingAgentPreserved === true &&
|
|
earlier.existingLocalSkillPreserved === true &&
|
|
(phase === "initial" ||
|
|
(later?.existingAgentPreserved === true &&
|
|
later.existingLocalSkillPreserved === true)),
|
|
"Actual accepted preview conflicts with a public-created unrelated agent, and the separately owned local skill really rejects catalog installation; both existing originals remain immutable across install and repeat.",
|
|
);
|
|
const resolved = (condition: Record<string, unknown> | undefined) => condition?.requiredCatalogStillMissing === false
|
|
&& verifiedTeamSkillRecovery(o.state, o.companyId, condition.recovery);
|
|
const accounted = (condition: Record<string, unknown> | undefined) => condition?.localSkillStillOccupiesSlug === true
|
|
&& (condition.requiredCatalogStillMissing === true || resolved(condition));
|
|
const events = o.activity as Array<Entity & { runId?: string | null }>;
|
|
c.add(
|
|
"actual-post-import-partial-warning",
|
|
!!target &&
|
|
accounted(earlier) &&
|
|
(phase === "initial" || accounted(later)) &&
|
|
events.some(
|
|
(e) =>
|
|
e.action === "company.team_catalog_installed" &&
|
|
e.actorType === "agent" &&
|
|
e.agentId === o.leadId &&
|
|
e.runId === o.state.initialRunId &&
|
|
e.details?.catalogId === (o.state.team as CatalogTeam)?.id &&
|
|
Number(e.details?.warningCount) > 0 &&
|
|
Number(e.details?.skillPreparationCount) > 0,
|
|
),
|
|
"The actual initial model's installer creates the team graph and reports a genuine post-import warning with the required skill either still blocked or independently recovered through its exact catalog version and intended agent bindings.",
|
|
);
|
|
// Reply meaning, including how it names the blocked skill, is left to human review. The case grades the
|
|
// partial outcome from saved state above (catalog skill still missing beside the local skill that holds
|
|
// its slug, plus the installer's own warning); the renamed member and project links are graded by the
|
|
// install case. A product-returned name is not required: models paraphrase slugs ("document-maintenance").
|
|
c.add(
|
|
"initial-reply-delivered",
|
|
!!target && first.trim().length > 0,
|
|
"The initial installer run delivers a reply in the conversation; its account of the blocked skill is read by a person.",
|
|
);
|
|
return c.result();
|
|
}
|