Files
PaperClipAI/server/src/services/onboarding-first-task-assets.test.ts
T
DottaandPaperclip 862a5758ba fix(agents): reduce hiring templates to role descriptions (#14985)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work.
> - New agents receive role instructions from onboarding, the hiring skill, or a team package.
> - These sources repeat harness procedures and impose generic work policies.
> - They can crowd out the task and the harness instructions.
> - This pull request reduces those sources to short role descriptions.
> - It preserves configuration, skills, authentication, reporting lines, and approval controls.
> - The benefit is less repeated instruction text with explicit coverage for the default hiring path.

## Linked Issues or Issue Description

Refs #3307. The CEO template can impose a fixed delegation route instead of letting the agent choose how to fulfill the request. This change removes that route. It does not implement autonomous goal selection.

Related work: #14920 preserves stock Codex base instructions. #14948 reduces the generic manual and shared runtime prompts. #14961 improves native completion-tool descriptions. This PR is separate from those changes.

## What Changed

- Select only the short CEO `AGENTS.md` for new default CEO bundles. Keep the three former companion files as compatibility assets.
- Reduce the first-agent chief-of-staff prompt and coder, QA, UX, and security role examples.
- Reduce seven bundled team role bodies. Preserve their role, reporting, and skill metadata. Regenerate the catalog.
- Make hiring examples optional. Replace the long generic role manual with short role drafting guidance. Preserve explicit requester instructions.
- Add configuration and import coverage for native and legacy managed bundles, custom instructions, first-agent rendering, and catalog contents.
- Add an explicit-only hiring eval that starts from the production CEO default and checks one coder hire, independently computed JSON output, saved instructions, and worker reuse.
- Include the full prompt comparison and a separate three-request drafting simulation. Neither is a live provider comparison.

Prompt differences: [before and after](doc/plans/2026-10-02-hiring-template-prompt-diff.md). The CEO default falls from 1,897 to 20 words. The coder example falls from 652 to 18 words. Word counts describe instruction size, not outcome quality or billing.

## Verification

- PASS: 99 focused server tests and eight shipped-catalog tests.
- PASS: catalog generation and validation for four shipped teams.
- PASS: hiring skill validation.
- PASS: `pnpm -r typecheck`.
- PASS: `pnpm build`.
- INCOMPLETE: the full local `pnpm test:run` was stopped before rebase. Its original log is retained. This is not a completed full-suite pass. The full current-head GitHub CI workflow passed: https://github.com/paperclipai/paperclip/actions/runs/37073372419.
- PASS: `pnpm test:e2e:runner:typecheck` and `pnpm test:e2e:runner:unit` (63 files / 843 tests).
- PASS: discovery for the two new hiring cells, 50 existing everyday cells, and the full 438-cell catalog.
- PASS after rebase: 99 server tests, 11 catalog tests, 62 selected E2E support tests, and the E2E typecheck.
- PASS: all current-head PR checks at `57dcee147ed0b2d2e3cc657cd9e50fb16bf9ec25`: 51 successful check runs, two intentional Storybook skips, and successful Snyk status. Fresh Greptile is 5/5 with zero unresolved threads.
- PENDING follow-up: matched live hiring runs on frozen integration refs. No live outcome-quality or non-regression result is claimed from the configuration checks or this merge.

The new suite has two local native cells: Codex and ACPX Claude. It expects five provider turns per cell. It compares source-derived bundles, so the historical long templates remain admissible. Missing successful source-read receipts make a pair uncomparable. They do not establish a behavior regression or equivalence.

## Risks

- New default roles have fewer prescribed procedures. Live checks must determine whether a removed instruction was needed for an outcome.
- Existing custom and saved bundles keep their contents. The retained companion assets avoid a source-file compatibility break.
- Specialized Summarizer, Reflection Coach, and Wiki Maintainer prompts remain unchanged. Their product contracts need separate review.
- The generic non-CEO fallback reduction is in #14948. This PR alone does not provide its eight-word fallback.
- Configuration tests and drafting simulations do not establish live outcome quality. QA, UX, security, and chief-of-staff hiring behavior remain outside the new two-cell comparison.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, code editing, shell tools, and delegated verification. The runtime does not expose the exact deployment model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-02 17:56:42 -05:00

120 lines
5.1 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { readFile } from "node:fs/promises";
import {
ONBOARDING_FIRST_TASK_OPENING_INTERVIEW_OPTION_ID,
ONBOARDING_FIRST_TASK_OPENING_QUESTION_ID,
ONBOARDING_FIRST_TASK_OPENING_TASK_OPTION_ID,
buildOnboardingFirstTaskBrief,
buildOnboardingFirstTaskOpeningQuestion,
buildOnboardingFirstAgentInstructionsBundle,
fillFirstTaskPlaceholders,
renderChiefOfStaffPersona,
renderOnboardingFirstTaskGreeting,
} from "./onboarding-first-task-assets.js";
describe("fillFirstTaskPlaceholders", () => {
it("fills the name and organization when present", () => {
const out = fillFirstTaskPlaceholders(
"I'm {{agentName}}, chief of staff for {{organizationName}}.",
{ agentName: "Ada", organizationName: "Acme" },
);
expect(out).toBe("I'm Ada, chief of staff for Acme.");
});
it("drops the name and its trailing separator when no name is set", () => {
const out = fillFirstTaskPlaceholders("I'm {{agentName}}, your first agent teammate.", {
agentName: null,
});
expect(out).toBe("I'm your first agent teammate.");
});
it("falls back to a generic organization label when missing", () => {
const out = fillFirstTaskPlaceholders("for {{organizationName}}.", {});
expect(out).toBe("for your organization.");
});
});
describe("renderOnboardingFirstTaskGreeting", () => {
it("renders the board-approved greeting with the agent name", async () => {
const greeting = await renderOnboardingFirstTaskGreeting({ agentName: "Ada" });
expect(greeting).toContain("Welcome to Paperclip! I'm Ada, your first agent teammate.");
// The "what would you like to do" question moved onto the opening card.
expect(greeting).not.toContain("What would you like to do?");
});
});
describe("buildOnboardingFirstTaskOpeningQuestion", () => {
it("builds the two-option opening card with a free-text task option", async () => {
const payload = await buildOnboardingFirstTaskOpeningQuestion();
expect(payload.version).toBe(1);
expect(payload.supersedeOnUserComment).toBe(true);
expect(payload.submitLabel).toBe("Continue");
expect(payload.questions).toHaveLength(1);
const [question] = payload.questions;
expect(question.id).toBe(ONBOARDING_FIRST_TASK_OPENING_QUESTION_ID);
expect(question.selectionMode).toBe("single");
expect(question.required).toBe(true);
expect(question.prompt).toBe("What would you like to do?");
expect(question.options.map((option) => option.id)).toEqual([
ONBOARDING_FIRST_TASK_OPENING_INTERVIEW_OPTION_ID,
ONBOARDING_FIRST_TASK_OPENING_TASK_OPTION_ID,
]);
expect(question.options[0].label).toBe(
"Interview me and propose a plan and an agent team to execute it.",
);
expect(question.options[0].freeText).toBeUndefined();
expect(question.options[1].label).toBe("I have a task in mind");
expect(question.options[1].freeText).toBe(true);
});
});
describe("buildOnboardingFirstTaskBrief", () => {
it.each([
{ usePlanProposal: false, mode: "confirmation" },
{ usePlanProposal: true, mode: "plan" },
])("invokes the skill with $mode mode without inlining the policy", async ({ usePlanProposal, mode }) => {
const brief = await buildOnboardingFirstTaskBrief({ usePlanProposal });
expect(brief).toContain("Use the `first-task` skill (/first-task)");
expect(brief).toContain("Read its SKILL.md");
expect(brief).toContain("subsequent wakes of this task");
expect(brief).toContain(`Single-task proposal mode: \`${mode}\`.`);
expect(brief).not.toContain("{{");
expect(brief).not.toContain("Take the path the user picked.");
expect(brief).not.toContain("request_confirmation");
expect(brief).not.toContain("request_checkbox_confirmation");
});
});
describe("first-task proposal mode policy", () => {
it("maps both persisted brief modes to their proposal forms", async () => {
const skill = await readFile(new URL("../onboarding-assets/first-task/skills/first-task/SKILL.md", import.meta.url), "utf8");
expect(skill).toContain("`confirmation` means one `request_confirmation`");
expect(skill).toContain("`plan` means save a short `plan` document");
expect(skill).toContain("`request_checkbox_confirmation` targeting its saved revision");
expect(skill).toContain("explicit plan requests regardless of the single-task proposal mode");
});
});
describe("chief-of-staff persona", () => {
it("renders the persona with placeholders filled", async () => {
const persona = await renderChiefOfStaffPersona({
agentName: "Ada",
organizationName: "Acme",
});
expect(persona).toContain("You are Ada, chief of staff for Acme.");
expect(persona).not.toContain("{{agentName}}");
expect(persona).not.toContain("{{organizationName}}");
});
it("returns an AGENTS.md-keyed bundle for the first agent", async () => {
const bundle = await buildOnboardingFirstAgentInstructionsBundle({
agentName: "Ada",
organizationName: "Acme",
});
expect(bundle.entryFile).toBe("AGENTS.md");
expect(bundle.files).toEqual({
"AGENTS.md": await renderChiefOfStaffPersona({ agentName: "Ada", organizationName: "Acme" }),
});
});
});