mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-11 05:31:46 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The pipeline subsystem models repeatable work as items moving through stages, with agent automation, review gates, blockers, drift notices, and linked work. > - Operators need this to be usable as one coherent workflow surface, not just as backend primitives or disconnected route experiments. > - The branch now carries the pipeline data model, service/routes, CLI/tutorial path, aggregation feeds, operator UI, stage automation controls, liveness/retry handling, and follow-up polish that make the primitive reviewable end to end. > - This pull request is the single review target for that pipeline workflow primitive stack. > - The benefit is that reviewers can evaluate the full operator experience and server contract together against `master`. ## Linked Issues or Issue Description No public GitHub issue exists for this work. The underlying feature request is described inline. ### Problem or motivation Paperclip needs a first-class way to model multi-stage agent/company workflows where upstream items can spawn downstream work, request review, carry fields across pipelines, surface drift, retry automation, and show operators where work is blocked or active. Without a unified pipeline primitive, these workflows spread across ad hoc issues, routines, and comments, making the state hard to inspect or operate. ### Proposed solution Add the pipeline workflow primitive stack: database schema and migrations, shared validators/types, server services and REST routes, aggregation and liveness helpers, CLI/tutorial smoke support, and the React operator UI for pipeline lists, boards, item detail, review/learnings views, settings, stage automation, secrets, carry-over fields, and retry/recovery flows. ### Alternatives considered - Keep workflows as loosely linked issues and routines: rejected because operators need a single board/detail/settings surface for repeated workflow patterns. - Ship backend primitives first and defer UI: rejected for this branch because the operator experience is the main way to validate the primitive. - Add a narrower one-off content workflow: rejected because the same primitives are useful across future company processes. ## What Changed - Added and evolved pipeline schema, migrations, shared contracts, server services, REST routes, route tests, and CLI/tutorial smoke support. - Added pipeline aggregation, health/liveness, drift acknowledgment, blocker/carry-over, automation retry, stage automation environment, and permission recovery behavior. - Added the operator UI for pipeline index/board/item detail/settings/review/learnings flows, including stage secrets, automation controls, markdown/item descriptions, linked issue assets, liveness banners, and source automation metadata. - Refactored issue document frame rendering through the shared `DocumentFrameHeader` component to keep document controls consistent with the pipeline document surfaces. - Kept this PR as the single base-branch review target for the current pipeline branch. ## Verification Current branch refresh: - `pnpm vitest run server/src/__tests__/pipelines-service.test.ts` — 31 passed - `pnpm vitest run server/src/__tests__/pipelines-routes.test.ts` — 19 passed - `pnpm --filter ./server typecheck` — passed - `pnpm --filter ./ui typecheck` — passed - Verified Pipelines remains gated by `enablePipelines === true`: sidebar item is hidden unless the flag is enabled, direct pipeline routes redirect to `/dashboard` when disabled, and the Experimental settings UI still has no Pipelines toggle. - GitHub status checks on `df071c710646de625131064c3fb6588b5e97964a` — all complete with no failing conclusions, including Actions, Socket, Superagent/Security, and Greptile Review - Greptile summary on `df071c710646de625131064c3fb6588b5e97964a` — Confidence Score 5/5 - GitHub review-thread sweep — 0 unresolved Greptile threads Previously recorded during branch development: - Server pipeline service/route and aggregation tests - Shared validator tests - UI pipeline page/settings/item-detail/learnings/liveness tests - Pipeline tutorial smoke path ## Risks - High review surface: this is a large feature branch spanning database, shared contracts, server behavior, CLI/docs, and UI. - Migration ordering and schema compatibility need reviewer attention because this branch has been kept current across multiple `master` syncs. - GitHub still reports merge state `BLOCKED` because the PR is awaiting normal human review/branch-protection completion; all current status checks are green. - Branch-name checklist exception: this PR uses the pre-existing requested branch name, which predates the current public-branch naming rule. The PR title/body avoid internal issue references. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex coding agent based on GPT-5, with repository tool use, shell execution, git/GitHub CLI operations, and local verification commands. Earlier commits in this branch were assisted by Paperclip agents and other AI coding agents as recorded in commit authorship. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
323 lines
9.5 KiB
TypeScript
323 lines
9.5 KiB
TypeScript
export const PROJECT_MENTION_SCHEME = "project://";
|
|
export const AGENT_MENTION_SCHEME = "agent://";
|
|
export const USER_MENTION_SCHEME = "user://";
|
|
export const SKILL_MENTION_SCHEME = "skill://";
|
|
export const ROUTINE_MENTION_SCHEME = "routine://";
|
|
export const PIPELINE_MENTION_SCHEME = "pipeline://";
|
|
|
|
const HEX_COLOR_RE = /^[0-9a-f]{6}$/i;
|
|
const HEX_COLOR_SHORT_RE = /^[0-9a-f]{3}$/i;
|
|
const HEX_COLOR_WITH_HASH_RE = /^#[0-9a-f]{6}$/i;
|
|
const HEX_COLOR_SHORT_WITH_HASH_RE = /^#[0-9a-f]{3}$/i;
|
|
const PROJECT_MENTION_LINK_RE = /\[[^\]]*]\((project:\/\/[^)\s]+)\)/gi;
|
|
const AGENT_MENTION_LINK_RE = /\[[^\]]*]\((agent:\/\/[^)\s]+)\)/gi;
|
|
const USER_MENTION_LINK_RE = /\[[^\]]*]\((user:\/\/[^)\s]+)\)/gi;
|
|
const SKILL_MENTION_LINK_RE = /\[[^\]]*]\((skill:\/\/[^)\s]+)\)/gi;
|
|
const ROUTINE_MENTION_LINK_RE = /\[[^\]]*]\((routine:\/\/[^)\s]+)\)/gi;
|
|
const PIPELINE_MENTION_LINK_RE = /\[[^\]]*]\((pipeline:\/\/[^)\s]+)\)/gi;
|
|
const AGENT_ICON_NAME_RE = /^[a-z0-9-]+$/i;
|
|
const SKILL_SLUG_RE = /^[a-z0-9][a-z0-9-]*$/i;
|
|
|
|
export interface ParsedProjectMention {
|
|
projectId: string;
|
|
color: string | null;
|
|
}
|
|
|
|
export interface ParsedAgentMention {
|
|
agentId: string;
|
|
icon: string | null;
|
|
}
|
|
|
|
export interface ParsedUserMention {
|
|
userId: string;
|
|
}
|
|
|
|
export interface ParsedSkillMention {
|
|
skillId: string;
|
|
slug: string | null;
|
|
}
|
|
|
|
export interface ParsedRoutineMention {
|
|
routineId: string;
|
|
}
|
|
|
|
export interface ParsedPipelineMention {
|
|
pipelineId: string;
|
|
stageKey: string | null;
|
|
}
|
|
|
|
function normalizeHexColor(input: string | null | undefined): string | null {
|
|
if (!input) return null;
|
|
const trimmed = input.trim();
|
|
if (!trimmed) return null;
|
|
|
|
if (HEX_COLOR_WITH_HASH_RE.test(trimmed)) {
|
|
return trimmed.toLowerCase();
|
|
}
|
|
if (HEX_COLOR_RE.test(trimmed)) {
|
|
return `#${trimmed.toLowerCase()}`;
|
|
}
|
|
if (HEX_COLOR_SHORT_WITH_HASH_RE.test(trimmed)) {
|
|
const raw = trimmed.slice(1).toLowerCase();
|
|
return `#${raw[0]}${raw[0]}${raw[1]}${raw[1]}${raw[2]}${raw[2]}`;
|
|
}
|
|
if (HEX_COLOR_SHORT_RE.test(trimmed)) {
|
|
const raw = trimmed.toLowerCase();
|
|
return `#${raw[0]}${raw[0]}${raw[1]}${raw[1]}${raw[2]}${raw[2]}`;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
export function buildProjectMentionHref(projectId: string, color?: string | null): string {
|
|
const trimmedProjectId = projectId.trim();
|
|
const normalizedColor = normalizeHexColor(color ?? null);
|
|
if (!normalizedColor) {
|
|
return `${PROJECT_MENTION_SCHEME}${trimmedProjectId}`;
|
|
}
|
|
return `${PROJECT_MENTION_SCHEME}${trimmedProjectId}?c=${encodeURIComponent(normalizedColor.slice(1))}`;
|
|
}
|
|
|
|
export function parseProjectMentionHref(href: string): ParsedProjectMention | null {
|
|
if (!href.startsWith(PROJECT_MENTION_SCHEME)) return null;
|
|
|
|
let url: URL;
|
|
try {
|
|
url = new URL(href);
|
|
} catch {
|
|
return null;
|
|
}
|
|
|
|
if (url.protocol !== "project:") return null;
|
|
|
|
const projectId = `${url.hostname}${url.pathname}`.replace(/^\/+/, "").trim();
|
|
if (!projectId) return null;
|
|
|
|
const color = normalizeHexColor(url.searchParams.get("c") ?? url.searchParams.get("color"));
|
|
|
|
return {
|
|
projectId,
|
|
color,
|
|
};
|
|
}
|
|
|
|
export function buildAgentMentionHref(agentId: string, icon?: string | null): string {
|
|
const trimmedAgentId = agentId.trim();
|
|
const normalizedIcon = normalizeAgentIcon(icon ?? null);
|
|
if (!normalizedIcon) {
|
|
return `${AGENT_MENTION_SCHEME}${trimmedAgentId}`;
|
|
}
|
|
return `${AGENT_MENTION_SCHEME}${trimmedAgentId}?i=${encodeURIComponent(normalizedIcon)}`;
|
|
}
|
|
|
|
export function parseAgentMentionHref(href: string): ParsedAgentMention | null {
|
|
if (!href.startsWith(AGENT_MENTION_SCHEME)) return null;
|
|
|
|
let url: URL;
|
|
try {
|
|
url = new URL(href);
|
|
} catch {
|
|
return null;
|
|
}
|
|
|
|
if (url.protocol !== "agent:") return null;
|
|
|
|
const agentId = `${url.hostname}${url.pathname}`.replace(/^\/+/, "").trim();
|
|
if (!agentId) return null;
|
|
|
|
return {
|
|
agentId,
|
|
icon: normalizeAgentIcon(url.searchParams.get("i") ?? url.searchParams.get("icon")),
|
|
};
|
|
}
|
|
|
|
export function buildUserMentionHref(userId: string): string {
|
|
return `${USER_MENTION_SCHEME}${userId.trim()}`;
|
|
}
|
|
|
|
export function parseUserMentionHref(href: string): ParsedUserMention | null {
|
|
if (!href.startsWith(USER_MENTION_SCHEME)) return null;
|
|
|
|
let url: URL;
|
|
try {
|
|
url = new URL(href);
|
|
} catch {
|
|
return null;
|
|
}
|
|
|
|
if (url.protocol !== "user:") return null;
|
|
|
|
const userId = `${url.hostname}${url.pathname}`.replace(/^\/+/, "").trim();
|
|
if (!userId) return null;
|
|
|
|
return { userId };
|
|
}
|
|
|
|
export function buildSkillMentionHref(skillId: string, slug?: string | null): string {
|
|
const trimmedSkillId = skillId.trim();
|
|
const normalizedSlug = normalizeSkillSlug(slug ?? null);
|
|
if (!normalizedSlug) {
|
|
return `${SKILL_MENTION_SCHEME}${trimmedSkillId}`;
|
|
}
|
|
return `${SKILL_MENTION_SCHEME}${trimmedSkillId}?s=${encodeURIComponent(normalizedSlug)}`;
|
|
}
|
|
|
|
export function parseSkillMentionHref(href: string): ParsedSkillMention | null {
|
|
if (!href.startsWith(SKILL_MENTION_SCHEME)) return null;
|
|
|
|
let url: URL;
|
|
try {
|
|
url = new URL(href);
|
|
} catch {
|
|
return null;
|
|
}
|
|
|
|
if (url.protocol !== "skill:") return null;
|
|
|
|
const skillId = `${url.hostname}${url.pathname}`.replace(/^\/+/, "").trim();
|
|
if (!skillId) return null;
|
|
|
|
return {
|
|
skillId,
|
|
slug: normalizeSkillSlug(url.searchParams.get("s") ?? url.searchParams.get("slug")),
|
|
};
|
|
}
|
|
|
|
export function buildRoutineMentionHref(routineId: string): string {
|
|
return `${ROUTINE_MENTION_SCHEME}${routineId.trim()}`;
|
|
}
|
|
|
|
export function parseRoutineMentionHref(href: string): ParsedRoutineMention | null {
|
|
if (!href.startsWith(ROUTINE_MENTION_SCHEME)) return null;
|
|
|
|
let url: URL;
|
|
try {
|
|
url = new URL(href);
|
|
} catch {
|
|
return null;
|
|
}
|
|
|
|
if (url.protocol !== "routine:") return null;
|
|
|
|
const routineId = `${url.hostname}${url.pathname}`.replace(/^\/+/, "").trim();
|
|
if (!routineId) return null;
|
|
|
|
return { routineId };
|
|
}
|
|
|
|
export function buildPipelineMentionHref(pipelineId: string, stageKey?: string | null): string {
|
|
const trimmedPipelineId = pipelineId.trim();
|
|
const normalizedStageKey = stageKey?.trim();
|
|
if (!normalizedStageKey) return `${PIPELINE_MENTION_SCHEME}${trimmedPipelineId}`;
|
|
return `${PIPELINE_MENTION_SCHEME}${trimmedPipelineId}?stage=${encodeURIComponent(normalizedStageKey)}`;
|
|
}
|
|
|
|
export function parsePipelineMentionHref(href: string): ParsedPipelineMention | null {
|
|
if (!href.startsWith(PIPELINE_MENTION_SCHEME)) return null;
|
|
|
|
let url: URL;
|
|
try {
|
|
url = new URL(href);
|
|
} catch {
|
|
return null;
|
|
}
|
|
|
|
if (url.protocol !== "pipeline:") return null;
|
|
|
|
const pipelineId = `${url.hostname}${url.pathname}`.replace(/^\/+/, "").trim();
|
|
if (!pipelineId) return null;
|
|
|
|
const stageKey = url.searchParams.get("stage")?.trim() || null;
|
|
return { pipelineId, stageKey };
|
|
}
|
|
|
|
export function extractProjectMentionIds(markdown: string): string[] {
|
|
if (!markdown) return [];
|
|
const ids = new Set<string>();
|
|
const re = new RegExp(PROJECT_MENTION_LINK_RE);
|
|
let match: RegExpExecArray | null;
|
|
while ((match = re.exec(markdown)) !== null) {
|
|
const parsed = parseProjectMentionHref(match[1]);
|
|
if (parsed) ids.add(parsed.projectId);
|
|
}
|
|
return [...ids];
|
|
}
|
|
|
|
export function extractAgentMentionIds(markdown: string): string[] {
|
|
if (!markdown) return [];
|
|
const ids = new Set<string>();
|
|
const re = new RegExp(AGENT_MENTION_LINK_RE);
|
|
let match: RegExpExecArray | null;
|
|
while ((match = re.exec(markdown)) !== null) {
|
|
const parsed = parseAgentMentionHref(match[1]);
|
|
if (parsed) ids.add(parsed.agentId);
|
|
}
|
|
return [...ids];
|
|
}
|
|
|
|
export function extractUserMentionIds(markdown: string): string[] {
|
|
if (!markdown) return [];
|
|
const ids = new Set<string>();
|
|
const re = new RegExp(USER_MENTION_LINK_RE);
|
|
let match: RegExpExecArray | null;
|
|
while ((match = re.exec(markdown)) !== null) {
|
|
const parsed = parseUserMentionHref(match[1]);
|
|
if (parsed) ids.add(parsed.userId);
|
|
}
|
|
return [...ids];
|
|
}
|
|
|
|
export function extractSkillMentionIds(markdown: string): string[] {
|
|
if (!markdown) return [];
|
|
const ids = new Set<string>();
|
|
const re = new RegExp(SKILL_MENTION_LINK_RE);
|
|
let match: RegExpExecArray | null;
|
|
while ((match = re.exec(markdown)) !== null) {
|
|
const parsed = parseSkillMentionHref(match[1]);
|
|
if (parsed) ids.add(parsed.skillId);
|
|
}
|
|
return [...ids];
|
|
}
|
|
|
|
export function extractRoutineMentionIds(markdown: string): string[] {
|
|
if (!markdown) return [];
|
|
const ids = new Set<string>();
|
|
const re = new RegExp(ROUTINE_MENTION_LINK_RE);
|
|
let match: RegExpExecArray | null;
|
|
while ((match = re.exec(markdown)) !== null) {
|
|
const parsed = parseRoutineMentionHref(match[1]);
|
|
if (parsed) ids.add(parsed.routineId);
|
|
}
|
|
return [...ids];
|
|
}
|
|
|
|
export function extractPipelineMentions(markdown: string): ParsedPipelineMention[] {
|
|
if (!markdown) return [];
|
|
const seen = new Set<string>();
|
|
const mentions: ParsedPipelineMention[] = [];
|
|
const re = new RegExp(PIPELINE_MENTION_LINK_RE);
|
|
let match: RegExpExecArray | null;
|
|
while ((match = re.exec(markdown)) !== null) {
|
|
const parsed = parsePipelineMentionHref(match[1]);
|
|
if (!parsed) continue;
|
|
const key = `${parsed.pipelineId}:${parsed.stageKey ?? ""}`;
|
|
if (seen.has(key)) continue;
|
|
seen.add(key);
|
|
mentions.push(parsed);
|
|
}
|
|
return mentions;
|
|
}
|
|
|
|
function normalizeAgentIcon(input: string | null | undefined): string | null {
|
|
if (!input) return null;
|
|
const trimmed = input.trim().toLowerCase();
|
|
if (!trimmed || !AGENT_ICON_NAME_RE.test(trimmed)) return null;
|
|
return trimmed;
|
|
}
|
|
|
|
function normalizeSkillSlug(input: string | null | undefined): string | null {
|
|
if (!input) return null;
|
|
const trimmed = input.trim().toLowerCase();
|
|
if (!trimmed || !SKILL_SLUG_RE.test(trimmed)) return null;
|
|
return trimmed;
|
|
}
|