mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 15:12:18 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The pipeline subsystem models repeatable work as items moving through stages, with agent automation, review gates, blockers, drift notices, and linked work. > - Operators need this to be usable as one coherent workflow surface, not just as backend primitives or disconnected route experiments. > - The branch now carries the pipeline data model, service/routes, CLI/tutorial path, aggregation feeds, operator UI, stage automation controls, liveness/retry handling, and follow-up polish that make the primitive reviewable end to end. > - This pull request is the single review target for that pipeline workflow primitive stack. > - The benefit is that reviewers can evaluate the full operator experience and server contract together against `master`. ## Linked Issues or Issue Description No public GitHub issue exists for this work. The underlying feature request is described inline. ### Problem or motivation Paperclip needs a first-class way to model multi-stage agent/company workflows where upstream items can spawn downstream work, request review, carry fields across pipelines, surface drift, retry automation, and show operators where work is blocked or active. Without a unified pipeline primitive, these workflows spread across ad hoc issues, routines, and comments, making the state hard to inspect or operate. ### Proposed solution Add the pipeline workflow primitive stack: database schema and migrations, shared validators/types, server services and REST routes, aggregation and liveness helpers, CLI/tutorial smoke support, and the React operator UI for pipeline lists, boards, item detail, review/learnings views, settings, stage automation, secrets, carry-over fields, and retry/recovery flows. ### Alternatives considered - Keep workflows as loosely linked issues and routines: rejected because operators need a single board/detail/settings surface for repeated workflow patterns. - Ship backend primitives first and defer UI: rejected for this branch because the operator experience is the main way to validate the primitive. - Add a narrower one-off content workflow: rejected because the same primitives are useful across future company processes. ## What Changed - Added and evolved pipeline schema, migrations, shared contracts, server services, REST routes, route tests, and CLI/tutorial smoke support. - Added pipeline aggregation, health/liveness, drift acknowledgment, blocker/carry-over, automation retry, stage automation environment, and permission recovery behavior. - Added the operator UI for pipeline index/board/item detail/settings/review/learnings flows, including stage secrets, automation controls, markdown/item descriptions, linked issue assets, liveness banners, and source automation metadata. - Refactored issue document frame rendering through the shared `DocumentFrameHeader` component to keep document controls consistent with the pipeline document surfaces. - Kept this PR as the single base-branch review target for the current pipeline branch. ## Verification Current branch refresh: - `pnpm vitest run server/src/__tests__/pipelines-service.test.ts` — 31 passed - `pnpm vitest run server/src/__tests__/pipelines-routes.test.ts` — 19 passed - `pnpm --filter ./server typecheck` — passed - `pnpm --filter ./ui typecheck` — passed - Verified Pipelines remains gated by `enablePipelines === true`: sidebar item is hidden unless the flag is enabled, direct pipeline routes redirect to `/dashboard` when disabled, and the Experimental settings UI still has no Pipelines toggle. - GitHub status checks on `df071c710646de625131064c3fb6588b5e97964a` — all complete with no failing conclusions, including Actions, Socket, Superagent/Security, and Greptile Review - Greptile summary on `df071c710646de625131064c3fb6588b5e97964a` — Confidence Score 5/5 - GitHub review-thread sweep — 0 unresolved Greptile threads Previously recorded during branch development: - Server pipeline service/route and aggregation tests - Shared validator tests - UI pipeline page/settings/item-detail/learnings/liveness tests - Pipeline tutorial smoke path ## Risks - High review surface: this is a large feature branch spanning database, shared contracts, server behavior, CLI/docs, and UI. - Migration ordering and schema compatibility need reviewer attention because this branch has been kept current across multiple `master` syncs. - GitHub still reports merge state `BLOCKED` because the PR is awaiting normal human review/branch-protection completion; all current status checks are green. - Branch-name checklist exception: this PR uses the pre-existing requested branch name, which predates the current public-branch naming rule. The PR title/body avoid internal issue references. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex coding agent based on GPT-5, with repository tool use, shell execution, git/GitHub CLI operations, and local verification commands. Earlier commits in this branch were assisted by Paperclip agents and other AI coding agents as recorded in commit authorship. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
194 lines
6.9 KiB
TypeScript
194 lines
6.9 KiB
TypeScript
import type { PipelineStage } from "../api/pipelines";
|
|
|
|
/**
|
|
* UI-side reader + copy helpers for the "Break into pieces" stage primitive.
|
|
*
|
|
* The server stores the breakdown config on `stage.config.breakdown` (see
|
|
* `pipelineStageBreakdownSchema`). Only the singular `pieceNoun` is persisted;
|
|
* the plural is derived here exactly the way the server's health checks derive
|
|
* it (`${pieceNoun}s`) so every count/banner string stays consistent.
|
|
*
|
|
* All copy in this module is prosumer-facing — no API terms ("case", "child",
|
|
* "stage key") ever surface; the configured piece noun is the dominant token.
|
|
*/
|
|
|
|
export interface StageBreakdownConfig {
|
|
targetPipelineId: string;
|
|
targetStageKey: string;
|
|
pieceNoun: string;
|
|
inheritFields: string[];
|
|
carryOverPolicy?: BreakdownCarryOverPolicy;
|
|
advanceTo: string | null;
|
|
waitForPieces: boolean;
|
|
whenFinishedMoveTo: string | null;
|
|
}
|
|
|
|
export type BreakdownCarryOverMode = "all_except" | "only";
|
|
|
|
export interface BreakdownCarryOverPolicy {
|
|
version: 1;
|
|
mode: BreakdownCarryOverMode;
|
|
includeFields: string[];
|
|
excludeFields: string[];
|
|
}
|
|
|
|
function asString(value: unknown): string {
|
|
return typeof value === "string" ? value.trim() : "";
|
|
}
|
|
|
|
function asStringList(value: unknown): string[] {
|
|
if (!Array.isArray(value)) return [];
|
|
const seen = new Set<string>();
|
|
return value.flatMap((entry) => {
|
|
const key = asString(entry);
|
|
if (!key || seen.has(key)) return [];
|
|
seen.add(key);
|
|
return [key];
|
|
});
|
|
}
|
|
|
|
function readCarryOverPolicy(record: Record<string, unknown>, inheritFields: string[]): BreakdownCarryOverPolicy {
|
|
const raw = record.carryOverPolicy;
|
|
if (raw && typeof raw === "object" && !Array.isArray(raw)) {
|
|
const policy = raw as Record<string, unknown>;
|
|
const mode = policy.mode === "all_except" || policy.mode === "only" ? policy.mode : "all_except";
|
|
return {
|
|
version: 1,
|
|
mode,
|
|
includeFields: asStringList(policy.includeFields),
|
|
excludeFields: asStringList(policy.excludeFields),
|
|
};
|
|
}
|
|
return {
|
|
version: 1,
|
|
mode: "only",
|
|
includeFields: inheritFields,
|
|
excludeFields: [],
|
|
};
|
|
}
|
|
|
|
export function isCarryOverIdentityFieldKey(key: string) {
|
|
const normalized = key.replace(/[^A-Za-z0-9]/g, "").toLowerCase();
|
|
return normalized === "name" ||
|
|
normalized === "title" ||
|
|
normalized === "casename" ||
|
|
normalized === "casetitle";
|
|
}
|
|
|
|
export function isCarryOverFieldEnabled(policy: BreakdownCarryOverPolicy | null | undefined, key: string) {
|
|
if (!policy) return false;
|
|
if (isCarryOverIdentityFieldKey(key)) return false;
|
|
if (policy.mode === "only") return policy.includeFields.includes(key);
|
|
return !policy.excludeFields.includes(key);
|
|
}
|
|
|
|
/**
|
|
* Returns the breakdown config when a stage has the `breakdown` block, else
|
|
* `null`. Fields may be empty when the config is half-finished — read surfaces
|
|
* lean on `computePipelineHealth` to flag a missing target rather than hiding
|
|
* the stage.
|
|
*/
|
|
export function readStageBreakdown(
|
|
stage: { config?: Record<string, unknown> | null } | null | undefined,
|
|
): StageBreakdownConfig | null {
|
|
const raw = stage?.config?.breakdown;
|
|
if (!raw || typeof raw !== "object" || Array.isArray(raw)) return null;
|
|
const record = raw as Record<string, unknown>;
|
|
const inheritFields = Array.isArray(record.inheritFields)
|
|
? record.inheritFields.filter((field): field is string => typeof field === "string" && field.trim().length > 0).map((field) => field.trim())
|
|
: [];
|
|
const carryOverPolicy = readCarryOverPolicy(record, inheritFields);
|
|
return {
|
|
targetPipelineId: asString(record.targetPipelineId),
|
|
targetStageKey: asString(record.targetStageKey),
|
|
pieceNoun: asString(record.pieceNoun) || "piece",
|
|
inheritFields,
|
|
carryOverPolicy,
|
|
advanceTo: asString(record.advanceTo) || null,
|
|
waitForPieces: record.waitForPieces === true,
|
|
whenFinishedMoveTo: asString(record.whenFinishedMoveTo) || null,
|
|
};
|
|
}
|
|
|
|
export function hasStageBreakdown(stage: PipelineStage | null | undefined): boolean {
|
|
return readStageBreakdown(stage) !== null;
|
|
}
|
|
|
|
/** Plural form of the piece noun, derived the same way the server does. */
|
|
export function pieceNounPlural(noun: string): string {
|
|
const trimmed = noun.trim() || "piece";
|
|
return `${trimmed}s`;
|
|
}
|
|
|
|
/** "a and b" / "a, b and c" — for inherited-field lists. */
|
|
export function joinWithAnd(items: string[]): string {
|
|
const list = items.filter((item) => item.trim().length > 0);
|
|
if (list.length === 0) return "";
|
|
if (list.length === 1) return list[0]!;
|
|
if (list.length === 2) return `${list[0]} and ${list[1]}`;
|
|
return `${list.slice(0, -1).join(", ")} and ${list[list.length - 1]}`;
|
|
}
|
|
|
|
export interface BreakdownCopyNames {
|
|
targetPipelineName: string;
|
|
entryStageName: string;
|
|
advanceToName: string | null;
|
|
whenFinishedName: string | null;
|
|
/** Human labels for the inherited fields, in config order. */
|
|
inheritedFieldLabels: string[];
|
|
}
|
|
|
|
/**
|
|
* The single generated sentence shown in the settings card footer band.
|
|
* Returns `null` when the config is too incomplete to summarize.
|
|
*/
|
|
export function breakdownSummarySentence(
|
|
config: StageBreakdownConfig,
|
|
names: BreakdownCopyNames,
|
|
): string | null {
|
|
if (!config.targetPipelineId || !config.targetStageKey || !names.targetPipelineName) {
|
|
return null;
|
|
}
|
|
const noun = config.pieceNoun;
|
|
const parts: string[] = [
|
|
`Paperclip will create one ${noun} per item in ${names.targetPipelineName} → ${names.entryStageName}`,
|
|
];
|
|
if (names.inheritedFieldLabels.length > 0) {
|
|
parts.push(`carry over ${joinWithAnd(names.inheritedFieldLabels)}`);
|
|
}
|
|
if (names.advanceToName) {
|
|
parts.push(`move this case to ${names.advanceToName}`);
|
|
}
|
|
let sentence = parts.join(", ");
|
|
if (config.waitForPieces && names.whenFinishedName) {
|
|
sentence += `, then wait until every ${noun} is finished before moving it to ${names.whenFinishedName}`;
|
|
}
|
|
return `${sentence}.`;
|
|
}
|
|
|
|
/**
|
|
* The read-only "Paperclip handles this" mechanics bullets, composed from the
|
|
* config. Bullets 5 and 6 only appear when the wait gate is on.
|
|
*/
|
|
export function breakdownMechanicsBullets(
|
|
config: StageBreakdownConfig,
|
|
names: BreakdownCopyNames,
|
|
): string[] {
|
|
const noun = config.pieceNoun;
|
|
const bullets: string[] = [
|
|
`Creates one ${noun} per item the agent returns, in ${names.targetPipelineName || "the destination pipeline"} → ${names.entryStageName || "its entry step"}.`,
|
|
`Links every ${noun} to this case so progress rolls up here.`,
|
|
];
|
|
if (names.inheritedFieldLabels.length > 0) {
|
|
bullets.push(`Carries over ${joinWithAnd(names.inheritedFieldLabels)} from this case onto each ${noun}.`);
|
|
}
|
|
if (names.advanceToName) {
|
|
bullets.push(`Moves this case to ${names.advanceToName} as soon as the pieces are created.`);
|
|
}
|
|
if (config.waitForPieces && names.whenFinishedName) {
|
|
bullets.push(`Waits until every ${noun} is finished, then moves this case to ${names.whenFinishedName}.`);
|
|
bullets.push(`If the agent returns an empty list, this case skips ahead to ${names.whenFinishedName}.`);
|
|
}
|
|
return bullets;
|
|
}
|