mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 16:35:48 +01:00
## Thinking Path > - Paperclip is the open source control plane people use to organize and operate AI-agent companies. > - Agent behavior depends partly on the bundled Paperclip core skill synchronized into each runtime. > - The existing database and runtime plumbing already supports immutable skill-version snapshots and per-agent version selections, but no product workflow exposed that capability. > - Replacing the live bundled skill globally would make champion adoption risky and difficult to compare across agents. > - This pull request adds an experimental, instance-level beta-skills gate plus a repository release registry, immutable seeded releases, enforcement, and a per-agent release picker. > - The benefit is controlled per-agent evaluation of frozen core-skill releases while the default-off path remains behaviorally unchanged. ## Linked Issues or Issue Description ### Subsystem affected Cross-cutting: `server/`, `ui/`, `packages/db`, and `packages/shared`. ### Problem or motivation Paperclip needs a safe way to evaluate improved versions of its core operating skill without globally replacing the live default. Today the version-snapshot and per-agent pin plumbing exists, but operators cannot use it. A global replacement would make regressions difficult to contain and would prevent controlled comparisons across agents. ### Proposed solution Add a default-off instance experiment that exposes immutable, named core-skill releases. When enabled, operators can pin each agent to a seeded release; when disabled, every agent resolves the live default while saved pins remain intact. Validate pinned writes at the API boundary, gate reads at runtime, and expose the selection in the agent Skills tab. ### Alternatives considered - **Replace the bundled core skill globally:** rejected because it changes every agent at once and provides no rollback/isolation boundary. - **Ship releases as separate skills:** rejected because releases are versions of one core capability, not independently enabled skills. - **Store release snapshots only outside the repository:** rejected because repository provenance and hashes make builds reproducible and reviewable. ### Roadmap alignment This extends the Skills Manager / Skill Studio direction in `ROADMAP.md` by making core-skill versions operable per agent. It does not duplicate another open implementation PR; GitHub searches found no related `enableBetaSkills` change. ### Additional context The feature remains experimental and default off. The V7 champion was selected through a multi-model evaluation process, and the frozen release contents are verified by SHA-256 below. ## What Changed - Added the default-off instance-level `enableBetaSkills` experimental flag. - Added `skills-releases/paperclip/` with the ordered release registry and frozen `v0` plus `v7-roster` snapshots. - Added release metadata to `company_skill_versions` and idempotent release seeding. The migration was planned as `0191`, then renumbered to `0192` because current `master` claimed `0191` before final rebase. - Added read-time gating and write-time validation so disabled instances always resolve the live default and reject pinned-version writes. - Added the per-agent Release picker in the agent Skills tab, including responsive layout and beta-pin state. - Kept `EDITS.md` out of the release registry and PR diff. ### V7 Adoption Evidence - Paid roster: 6 models, 94-case suite. - Result: 553/564 pass-within-2, mean 92.17/94, versus the P2 baseline of 544/564. - Reference model improved 84→91; maximin improved 84→90. - Final report: https://pages.paperclip.ing/skills/optimization/paperclip/pap-14624-p3-final-20260721/ ### Provenance - `v7-roster` is the Phase 1 champion plus additions-only edits E107–E112. Per-edit rationale remains in the evals repository at `source/v7-roster/EDITS.md` and is deliberately excluded from this PR. - `v0` is the `skills/paperclip` tree from commit `ea66ea81`. - Champion selection was accepted on July 21, 2026 via board card `9c304fc2` (PAP-14624 G3). - This delivery mechanism was accepted on July 24, 2026 via plan revision `2367abd2` (PAP-14858). ### QA Evidence - P4 QA matrix comment `b7f40522-4e9b-4a3a-9821-28e86fe1a987`: all 6 acceptance criteria passed. - Automated QA matrix: 166 tests passed with 0 failures, including real filesystem materialization and full SHA-256 assertions. - UI QA exercised the real agent Skills tab at desktop and mobile widths with the experimental flag both on and off. ## Verification - `pnpm check:token-gates` - Focused beta-release matrix: 169 tests passed across shared validators, server services/routes/heartbeat behavior, instance settings UI, and release picker UI. - `pnpm -r typecheck` - `pnpm build` - `pnpm test:run`: server and UI partitions passed; one CLI doctor test inherited temporary AWS credentials from the agent heartbeat and expected no static credentials. The isolated rerun with `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` unset passed 8/8. - V7 SHA-256: - `SKILL.md`: `53ab290489684cbf116fdd1406a95f6b6f53c9c36358b1bf8bfeae481e253575` - `references/cases.md`: `3b821f59064a7761091020a14819a8d787131f24029748563d6c0e1be7e6eaec` - `references/workflows.md`: `69747bd6e05f7e3673d1e67b07ff295df1869c05e1fd029804d5fa9177db92cd` - Confirmed 49 changed files, no `pnpm-lock.yaml`, no workflow changes, and no `EDITS.md`. ## Risks - **Migration:** low-to-moderate risk. Three nullable columns and one partial unique index are added idempotently; existing rows remain valid. - **Behavior:** low risk while the flag is off because read-time resolution forces the live default and saved pins are preserved but inactive. - **Frozen content:** release snapshots intentionally diverge from future live skill edits; provenance and hashes make that divergence explicit and reproducible. - **UI:** low risk. The picker only renders for the bundled core skill when the experimental flag is enabled and seeded releases exist. > This extends the existing Skills Manager / Skill Studio direction described in `ROADMAP.md`; it does not duplicate another open implementation PR. The GitHub PR search found no related `enableBetaSkills` change. ## Model Used - OpenAI Codex using `gpt-5.5` with reasoning and terminal/code-execution tools; context-window size is not exposed by this runtime. Earlier implementation commits also record Claude Opus 4.8 assistance where applicable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [ ] I have not referenced internal/instance-local Paperclip issues or links (required governance identifiers are included above; no internal URL is included) - [ ] My branch name describes the change and contains no internal Paperclip ticket id (the approved delivery plan mandated this shared branch name) - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
351 lines
12 KiB
TypeScript
351 lines
12 KiB
TypeScript
// @vitest-environment jsdom
|
|
|
|
import type { ReactNode } from "react";
|
|
import { flushSync } from "react-dom";
|
|
import { createRoot, type Root } from "react-dom/client";
|
|
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
|
import type {
|
|
AgentDesiredSkillEntry,
|
|
CompanySkillDetail,
|
|
CompanySkillUsageAgent,
|
|
CompanySkillVersion,
|
|
} from "@paperclipai/shared";
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
AgentsUsingSkillBadge,
|
|
AgentsUsingSkillDialog,
|
|
} from "./AgentsUsingSkillDialog";
|
|
|
|
const mockAgentsApi = vi.hoisted(() => ({
|
|
list: vi.fn(),
|
|
skills: vi.fn(),
|
|
syncSkills: vi.fn(),
|
|
}));
|
|
const mockCompanySkillsApi = vi.hoisted(() => ({
|
|
versions: vi.fn(),
|
|
}));
|
|
const mockPushToast = vi.hoisted(() => vi.fn());
|
|
|
|
vi.mock("@/lib/router", () => ({
|
|
Link: ({ children, to }: { children: ReactNode; to: string }) => <a href={to}>{children}</a>,
|
|
}));
|
|
vi.mock("@/api/agents", () => ({ agentsApi: mockAgentsApi }));
|
|
vi.mock("@/api/companySkills", () => ({ companySkillsApi: mockCompanySkillsApi }));
|
|
vi.mock("@/adapters/use-adapter-capabilities", () => ({
|
|
useAdapterCapabilities: () => () => ({
|
|
supportsInstructionsBundle: true,
|
|
supportsSkills: true,
|
|
supportsLocalAgentJwt: true,
|
|
requiresMaterializedRuntimeSkills: false,
|
|
supportsModelProfiles: false,
|
|
}),
|
|
}));
|
|
vi.mock("@/context/ToastContext", () => ({
|
|
useOptionalToastActions: () => ({ pushToast: mockPushToast }),
|
|
}));
|
|
|
|
(globalThis as unknown as { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
|
|
|
|
let root: Root | null = null;
|
|
let container: HTMLDivElement | null = null;
|
|
|
|
// React's `act` is not usable in this vitest setup (known gotcha); drive
|
|
// updates through flushSync like SkillStudio.test.tsx does.
|
|
async function act(callback: () => void | Promise<void>) {
|
|
let result: void | Promise<void> = undefined;
|
|
flushSync(() => {
|
|
result = callback();
|
|
});
|
|
await result;
|
|
}
|
|
|
|
async function flush() {
|
|
await Promise.resolve();
|
|
await new Promise((resolve) => window.setTimeout(resolve, 0));
|
|
}
|
|
|
|
function teardown() {
|
|
if (root) flushSync(() => root?.unmount());
|
|
container?.remove();
|
|
root = null;
|
|
container = null;
|
|
// Radix Dialog portals to document.body; drop any leftovers between renders.
|
|
document.body.innerHTML = "";
|
|
}
|
|
|
|
async function renderNode(node: ReactNode) {
|
|
teardown();
|
|
container = document.createElement("div");
|
|
document.body.appendChild(container);
|
|
root = createRoot(container);
|
|
const queryClient = new QueryClient({
|
|
defaultOptions: { queries: { retry: false }, mutations: { retry: false } },
|
|
});
|
|
await act(async () => {
|
|
root?.render(<QueryClientProvider client={queryClient}>{node}</QueryClientProvider>);
|
|
});
|
|
await flush();
|
|
}
|
|
|
|
async function click(el: Element) {
|
|
await act(async () => {
|
|
el.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
|
});
|
|
await flush();
|
|
}
|
|
|
|
function findButton(name: string): HTMLButtonElement {
|
|
const match = Array.from(document.body.querySelectorAll("button")).find(
|
|
(button) => button.textContent?.trim() === name,
|
|
) as HTMLButtonElement | undefined;
|
|
if (!match) throw new Error(`No button labelled "${name}"`);
|
|
return match;
|
|
}
|
|
|
|
function makeAgent(overrides: Partial<CompanySkillUsageAgent> = {}): CompanySkillUsageAgent {
|
|
return {
|
|
id: "agent-1",
|
|
name: "Reviewer",
|
|
urlKey: "reviewer",
|
|
adapterType: "claude_local",
|
|
desired: true,
|
|
actualState: null,
|
|
versionId: null,
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
function makeVersion(overrides: Partial<CompanySkillVersion> = {}): CompanySkillVersion {
|
|
return {
|
|
id: "ver-1",
|
|
companyId: "company-1",
|
|
companySkillId: "skill-1",
|
|
revisionNumber: 1,
|
|
label: null,
|
|
releaseId: null,
|
|
releaseName: null,
|
|
releasedAt: null,
|
|
fileInventory: [],
|
|
authorAgentId: null,
|
|
authorUserId: null,
|
|
createdAt: new Date("2026-01-01T00:00:00Z"),
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
function makeSkill(overrides: Partial<CompanySkillDetail> = {}): CompanySkillDetail {
|
|
return {
|
|
id: "skill-1",
|
|
companyId: "company-1",
|
|
key: "paperclip/demo",
|
|
slug: "demo",
|
|
name: "Demo Skill",
|
|
description: null,
|
|
markdown: "",
|
|
sourceType: "local_path",
|
|
sourceLocator: null,
|
|
sourceRef: null,
|
|
trustLevel: "markdown_only",
|
|
compatibility: "compatible",
|
|
fileInventory: [],
|
|
iconUrl: null,
|
|
color: null,
|
|
tagline: null,
|
|
authorName: null,
|
|
homepageUrl: null,
|
|
categories: [],
|
|
sharingScope: "company",
|
|
publicShareToken: null,
|
|
forkedFromSkillId: null,
|
|
forkedFromCompanyId: null,
|
|
starCount: 0,
|
|
installCount: 0,
|
|
forkCount: 0,
|
|
currentVersionId: "ver-3",
|
|
metadata: null,
|
|
createdAt: new Date("2026-01-01T00:00:00Z"),
|
|
updatedAt: new Date("2026-01-02T00:00:00Z"),
|
|
attachedAgentCount: 0,
|
|
usedByAgents: [],
|
|
editable: true,
|
|
editableReason: null,
|
|
sourceLabel: "Local",
|
|
sourceBadge: "local",
|
|
sourcePath: null,
|
|
currentVersion: makeVersion({ id: "ver-3", revisionNumber: 3 }),
|
|
starredByCurrentActor: false,
|
|
existingForks: [],
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
beforeEach(() => {
|
|
mockAgentsApi.list.mockResolvedValue([]);
|
|
mockAgentsApi.skills.mockResolvedValue({
|
|
adapterType: "claude_local",
|
|
supported: true,
|
|
mode: "managed",
|
|
desiredSkills: [],
|
|
desiredSkillEntries: [],
|
|
entries: [],
|
|
warnings: [],
|
|
});
|
|
mockAgentsApi.syncSkills.mockResolvedValue({});
|
|
mockCompanySkillsApi.versions.mockResolvedValue([]);
|
|
mockPushToast.mockReset();
|
|
});
|
|
|
|
afterEach(() => {
|
|
teardown();
|
|
vi.clearAllMocks();
|
|
});
|
|
|
|
describe("AgentsUsingSkillBadge", () => {
|
|
it("renders a muted '0 agents' button with an a11y label when unused", async () => {
|
|
await renderNode(<AgentsUsingSkillBadge companyId="company-1" skill={makeSkill()} />);
|
|
const badge = findButton("0 agents");
|
|
expect(badge).toBeDefined();
|
|
expect(badge?.getAttribute("aria-label")).toBe("0 agents use this skill");
|
|
expect(badge?.className).toContain("text-muted-foreground");
|
|
});
|
|
|
|
it("pluralizes and opens the dialog on click", async () => {
|
|
const skill = makeSkill({
|
|
usedByAgents: [makeAgent(), makeAgent({ id: "agent-2", name: "Coder", urlKey: "coder" })],
|
|
});
|
|
await renderNode(<AgentsUsingSkillBadge companyId="company-1" skill={skill} />);
|
|
const badge = findButton("2 agents");
|
|
expect(badge?.getAttribute("aria-label")).toBe("2 agents use this skill");
|
|
await click(badge!);
|
|
expect(document.body.textContent).toContain("Agents using Demo Skill");
|
|
expect(document.body.textContent).toContain("Reviewer");
|
|
expect(document.body.textContent).toContain("Coder");
|
|
});
|
|
});
|
|
|
|
describe("AgentsUsingSkillDialog", () => {
|
|
it("shows a version select with Latest (vN) and a static '—' when no version history", async () => {
|
|
mockCompanySkillsApi.versions.mockResolvedValue([
|
|
makeVersion({ id: "ver-3", revisionNumber: 3 }),
|
|
makeVersion({ id: "ver-2", revisionNumber: 2 }),
|
|
]);
|
|
const skill = makeSkill({ usedByAgents: [makeAgent()] });
|
|
await renderNode(
|
|
<AgentsUsingSkillDialog open onOpenChange={() => {}} companyId="company-1" skill={skill} />,
|
|
);
|
|
const select = document.body.querySelector("select");
|
|
expect(select).not.toBeNull();
|
|
expect(select?.textContent).toContain("Latest (v3)");
|
|
expect(select?.textContent).toContain("v2");
|
|
|
|
// No version history → "—" and no select.
|
|
await renderNode(
|
|
<AgentsUsingSkillDialog
|
|
open
|
|
onOpenChange={() => {}}
|
|
companyId="company-1"
|
|
skill={makeSkill({ currentVersion: null, currentVersionId: null, usedByAgents: [makeAgent()] })}
|
|
/>,
|
|
);
|
|
// The second render's dialog is the last-opened; its row shows an em dash.
|
|
expect(document.body.textContent).toContain("—");
|
|
});
|
|
|
|
it("shows a 'behind latest' hint for a stale pin", async () => {
|
|
mockCompanySkillsApi.versions.mockResolvedValue([
|
|
makeVersion({ id: "ver-3", revisionNumber: 3 }),
|
|
makeVersion({ id: "ver-1", revisionNumber: 1 }),
|
|
]);
|
|
const skill = makeSkill({ usedByAgents: [makeAgent({ versionId: "ver-1" })] });
|
|
await renderNode(
|
|
<AgentsUsingSkillDialog open onOpenChange={() => {}} companyId="company-1" skill={skill} />,
|
|
);
|
|
expect(document.body.textContent).toContain("2 versions behind latest");
|
|
});
|
|
|
|
it("removes via GET-then-sync and preserves the agent's other skills", async () => {
|
|
mockAgentsApi.skills.mockResolvedValue({
|
|
adapterType: "claude_local",
|
|
supported: true,
|
|
mode: "managed",
|
|
desiredSkills: ["paperclip/demo", "paperclip/other"],
|
|
desiredSkillEntries: [
|
|
{ key: "paperclip/demo", versionId: null },
|
|
{ key: "paperclip/other", versionId: null },
|
|
] satisfies AgentDesiredSkillEntry[],
|
|
entries: [],
|
|
warnings: [],
|
|
});
|
|
const skill = makeSkill({ usedByAgents: [makeAgent()] });
|
|
await renderNode(
|
|
<AgentsUsingSkillDialog open onOpenChange={() => {}} companyId="company-1" skill={skill} />,
|
|
);
|
|
|
|
// The trash button has no text; grab it by aria-label.
|
|
const trash = document.body.querySelector<HTMLButtonElement>(
|
|
'button[aria-label^="Remove this skill from"]',
|
|
);
|
|
await click(trash!);
|
|
const confirm = document.body.querySelector<HTMLButtonElement>(
|
|
'button[aria-label^="Confirm removing this skill from"]',
|
|
);
|
|
await click(confirm!);
|
|
|
|
expect(mockAgentsApi.skills).toHaveBeenCalledWith("agent-1", "company-1");
|
|
expect(mockAgentsApi.syncSkills).toHaveBeenCalledTimes(1);
|
|
const [, sentEntries] = mockAgentsApi.syncSkills.mock.calls[0];
|
|
expect(sentEntries).toEqual([{ key: "paperclip/other", versionId: null }]);
|
|
});
|
|
|
|
it("pins a version by sending the full set with the target repinned", async () => {
|
|
mockCompanySkillsApi.versions.mockResolvedValue([
|
|
makeVersion({ id: "ver-3", revisionNumber: 3 }),
|
|
makeVersion({ id: "ver-2", revisionNumber: 2 }),
|
|
]);
|
|
mockAgentsApi.skills.mockResolvedValue({
|
|
adapterType: "claude_local",
|
|
supported: true,
|
|
mode: "managed",
|
|
desiredSkills: ["paperclip/demo", "paperclip/other"],
|
|
desiredSkillEntries: [
|
|
{ key: "paperclip/demo", versionId: null },
|
|
{ key: "paperclip/other", versionId: null },
|
|
] satisfies AgentDesiredSkillEntry[],
|
|
entries: [],
|
|
warnings: [],
|
|
});
|
|
const skill = makeSkill({ usedByAgents: [makeAgent()] });
|
|
await renderNode(
|
|
<AgentsUsingSkillDialog open onOpenChange={() => {}} companyId="company-1" skill={skill} />,
|
|
);
|
|
const select = document.body.querySelector<HTMLSelectElement>("select");
|
|
await act(async () => {
|
|
const setter = Object.getOwnPropertyDescriptor(HTMLSelectElement.prototype, "value")?.set;
|
|
setter?.call(select, "ver-2");
|
|
select?.dispatchEvent(new Event("change", { bubbles: true }));
|
|
});
|
|
await flush();
|
|
|
|
expect(mockAgentsApi.syncSkills).toHaveBeenCalledTimes(1);
|
|
const [, sentEntries] = mockAgentsApi.syncSkills.mock.calls[0];
|
|
expect(sentEntries).toContainEqual({ key: "paperclip/other", versionId: null });
|
|
expect(sentEntries).toContainEqual({ key: "paperclip/demo", versionId: "ver-2" });
|
|
});
|
|
|
|
it("hides mutating controls in read-only mode but keeps the roster", async () => {
|
|
const skill = makeSkill({ usedByAgents: [makeAgent()] });
|
|
await renderNode(
|
|
<AgentsUsingSkillDialog
|
|
open
|
|
onOpenChange={() => {}}
|
|
companyId="company-1"
|
|
skill={skill}
|
|
canManage={false}
|
|
/>,
|
|
);
|
|
expect(document.body.textContent).toContain("Reviewer");
|
|
expect(document.body.querySelector('button[aria-label^="Remove this skill from"]')).toBeNull();
|
|
expect(document.body.querySelector("select")).toBeNull();
|
|
});
|
|
});
|