Files
PaperClipAI/ui/src/lib/pipeline-breakdown.ts
T
DottaandPaperclip 43b005b704 Add pipeline workflow primitives and operator UI (#7903)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The pipeline subsystem models repeatable work as items moving
through stages, with agent automation, review gates, blockers, drift
notices, and linked work.
> - Operators need this to be usable as one coherent workflow surface,
not just as backend primitives or disconnected route experiments.
> - The branch now carries the pipeline data model, service/routes,
CLI/tutorial path, aggregation feeds, operator UI, stage automation
controls, liveness/retry handling, and follow-up polish that make the
primitive reviewable end to end.
> - This pull request is the single review target for that pipeline
workflow primitive stack.
> - The benefit is that reviewers can evaluate the full operator
experience and server contract together against `master`.

## Linked Issues or Issue Description

No public GitHub issue exists for this work. The underlying feature
request is described inline.

### Problem or motivation

Paperclip needs a first-class way to model multi-stage agent/company
workflows where upstream items can spawn downstream work, request
review, carry fields across pipelines, surface drift, retry automation,
and show operators where work is blocked or active. Without a unified
pipeline primitive, these workflows spread across ad hoc issues,
routines, and comments, making the state hard to inspect or operate.

### Proposed solution

Add the pipeline workflow primitive stack: database schema and
migrations, shared validators/types, server services and REST routes,
aggregation and liveness helpers, CLI/tutorial smoke support, and the
React operator UI for pipeline lists, boards, item detail,
review/learnings views, settings, stage automation, secrets, carry-over
fields, and retry/recovery flows.

### Alternatives considered

- Keep workflows as loosely linked issues and routines: rejected because
operators need a single board/detail/settings surface for repeated
workflow patterns.
- Ship backend primitives first and defer UI: rejected for this branch
because the operator experience is the main way to validate the
primitive.
- Add a narrower one-off content workflow: rejected because the same
primitives are useful across future company processes.

## What Changed

- Added and evolved pipeline schema, migrations, shared contracts,
server services, REST routes, route tests, and CLI/tutorial smoke
support.
- Added pipeline aggregation, health/liveness, drift acknowledgment,
blocker/carry-over, automation retry, stage automation environment, and
permission recovery behavior.
- Added the operator UI for pipeline index/board/item
detail/settings/review/learnings flows, including stage secrets,
automation controls, markdown/item descriptions, linked issue assets,
liveness banners, and source automation metadata.
- Refactored issue document frame rendering through the shared
`DocumentFrameHeader` component to keep document controls consistent
with the pipeline document surfaces.
- Kept this PR as the single base-branch review target for the current
pipeline branch.

## Verification

Current branch refresh:

- `pnpm vitest run server/src/__tests__/pipelines-service.test.ts` — 31
passed
- `pnpm vitest run server/src/__tests__/pipelines-routes.test.ts` — 19
passed
- `pnpm --filter ./server typecheck` — passed
- `pnpm --filter ./ui typecheck` — passed
- Verified Pipelines remains gated by `enablePipelines === true`:
sidebar item is hidden unless the flag is enabled, direct pipeline
routes redirect to `/dashboard` when disabled, and the Experimental
settings UI still has no Pipelines toggle.
- GitHub status checks on `df071c710646de625131064c3fb6588b5e97964a` —
all complete with no failing conclusions, including Actions, Socket,
Superagent/Security, and Greptile Review
- Greptile summary on `df071c710646de625131064c3fb6588b5e97964a` —
Confidence Score 5/5
- GitHub review-thread sweep — 0 unresolved Greptile threads

Previously recorded during branch development:

- Server pipeline service/route and aggregation tests
- Shared validator tests
- UI pipeline page/settings/item-detail/learnings/liveness tests
- Pipeline tutorial smoke path

## Risks

- High review surface: this is a large feature branch spanning database,
shared contracts, server behavior, CLI/docs, and UI.
- Migration ordering and schema compatibility need reviewer attention
because this branch has been kept current across multiple `master`
syncs.
- GitHub still reports merge state `BLOCKED` because the PR is awaiting
normal human review/branch-protection completion; all current status
checks are green.
- Branch-name checklist exception: this PR uses the pre-existing
requested branch name, which predates the current public-branch naming
rule. The PR title/body avoid internal issue references.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex coding agent based on GPT-5, with repository tool use,
shell execution, git/GitHub CLI operations, and local verification
commands. Earlier commits in this branch were assisted by Paperclip
agents and other AI coding agents as recorded in commit authorship.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-26 12:02:44 -05:00

194 lines
6.9 KiB
TypeScript

import type { PipelineStage } from "../api/pipelines";
/**
* UI-side reader + copy helpers for the "Break into pieces" stage primitive.
*
* The server stores the breakdown config on `stage.config.breakdown` (see
* `pipelineStageBreakdownSchema`). Only the singular `pieceNoun` is persisted;
* the plural is derived here exactly the way the server's health checks derive
* it (`${pieceNoun}s`) so every count/banner string stays consistent.
*
* All copy in this module is prosumer-facing — no API terms ("case", "child",
* "stage key") ever surface; the configured piece noun is the dominant token.
*/
export interface StageBreakdownConfig {
targetPipelineId: string;
targetStageKey: string;
pieceNoun: string;
inheritFields: string[];
carryOverPolicy?: BreakdownCarryOverPolicy;
advanceTo: string | null;
waitForPieces: boolean;
whenFinishedMoveTo: string | null;
}
export type BreakdownCarryOverMode = "all_except" | "only";
export interface BreakdownCarryOverPolicy {
version: 1;
mode: BreakdownCarryOverMode;
includeFields: string[];
excludeFields: string[];
}
function asString(value: unknown): string {
return typeof value === "string" ? value.trim() : "";
}
function asStringList(value: unknown): string[] {
if (!Array.isArray(value)) return [];
const seen = new Set<string>();
return value.flatMap((entry) => {
const key = asString(entry);
if (!key || seen.has(key)) return [];
seen.add(key);
return [key];
});
}
function readCarryOverPolicy(record: Record<string, unknown>, inheritFields: string[]): BreakdownCarryOverPolicy {
const raw = record.carryOverPolicy;
if (raw && typeof raw === "object" && !Array.isArray(raw)) {
const policy = raw as Record<string, unknown>;
const mode = policy.mode === "all_except" || policy.mode === "only" ? policy.mode : "all_except";
return {
version: 1,
mode,
includeFields: asStringList(policy.includeFields),
excludeFields: asStringList(policy.excludeFields),
};
}
return {
version: 1,
mode: "only",
includeFields: inheritFields,
excludeFields: [],
};
}
export function isCarryOverIdentityFieldKey(key: string) {
const normalized = key.replace(/[^A-Za-z0-9]/g, "").toLowerCase();
return normalized === "name" ||
normalized === "title" ||
normalized === "casename" ||
normalized === "casetitle";
}
export function isCarryOverFieldEnabled(policy: BreakdownCarryOverPolicy | null | undefined, key: string) {
if (!policy) return false;
if (isCarryOverIdentityFieldKey(key)) return false;
if (policy.mode === "only") return policy.includeFields.includes(key);
return !policy.excludeFields.includes(key);
}
/**
* Returns the breakdown config when a stage has the `breakdown` block, else
* `null`. Fields may be empty when the config is half-finished — read surfaces
* lean on `computePipelineHealth` to flag a missing target rather than hiding
* the stage.
*/
export function readStageBreakdown(
stage: { config?: Record<string, unknown> | null } | null | undefined,
): StageBreakdownConfig | null {
const raw = stage?.config?.breakdown;
if (!raw || typeof raw !== "object" || Array.isArray(raw)) return null;
const record = raw as Record<string, unknown>;
const inheritFields = Array.isArray(record.inheritFields)
? record.inheritFields.filter((field): field is string => typeof field === "string" && field.trim().length > 0).map((field) => field.trim())
: [];
const carryOverPolicy = readCarryOverPolicy(record, inheritFields);
return {
targetPipelineId: asString(record.targetPipelineId),
targetStageKey: asString(record.targetStageKey),
pieceNoun: asString(record.pieceNoun) || "piece",
inheritFields,
carryOverPolicy,
advanceTo: asString(record.advanceTo) || null,
waitForPieces: record.waitForPieces === true,
whenFinishedMoveTo: asString(record.whenFinishedMoveTo) || null,
};
}
export function hasStageBreakdown(stage: PipelineStage | null | undefined): boolean {
return readStageBreakdown(stage) !== null;
}
/** Plural form of the piece noun, derived the same way the server does. */
export function pieceNounPlural(noun: string): string {
const trimmed = noun.trim() || "piece";
return `${trimmed}s`;
}
/** "a and b" / "a, b and c" — for inherited-field lists. */
export function joinWithAnd(items: string[]): string {
const list = items.filter((item) => item.trim().length > 0);
if (list.length === 0) return "";
if (list.length === 1) return list[0]!;
if (list.length === 2) return `${list[0]} and ${list[1]}`;
return `${list.slice(0, -1).join(", ")} and ${list[list.length - 1]}`;
}
export interface BreakdownCopyNames {
targetPipelineName: string;
entryStageName: string;
advanceToName: string | null;
whenFinishedName: string | null;
/** Human labels for the inherited fields, in config order. */
inheritedFieldLabels: string[];
}
/**
* The single generated sentence shown in the settings card footer band.
* Returns `null` when the config is too incomplete to summarize.
*/
export function breakdownSummarySentence(
config: StageBreakdownConfig,
names: BreakdownCopyNames,
): string | null {
if (!config.targetPipelineId || !config.targetStageKey || !names.targetPipelineName) {
return null;
}
const noun = config.pieceNoun;
const parts: string[] = [
`Paperclip will create one ${noun} per item in ${names.targetPipelineName} → ${names.entryStageName}`,
];
if (names.inheritedFieldLabels.length > 0) {
parts.push(`carry over ${joinWithAnd(names.inheritedFieldLabels)}`);
}
if (names.advanceToName) {
parts.push(`move this case to ${names.advanceToName}`);
}
let sentence = parts.join(", ");
if (config.waitForPieces && names.whenFinishedName) {
sentence += `, then wait until every ${noun} is finished before moving it to ${names.whenFinishedName}`;
}
return `${sentence}.`;
}
/**
* The read-only "Paperclip handles this" mechanics bullets, composed from the
* config. Bullets 5 and 6 only appear when the wait gate is on.
*/
export function breakdownMechanicsBullets(
config: StageBreakdownConfig,
names: BreakdownCopyNames,
): string[] {
const noun = config.pieceNoun;
const bullets: string[] = [
`Creates one ${noun} per item the agent returns, in ${names.targetPipelineName || "the destination pipeline"} → ${names.entryStageName || "its entry step"}.`,
`Links every ${noun} to this case so progress rolls up here.`,
];
if (names.inheritedFieldLabels.length > 0) {
bullets.push(`Carries over ${joinWithAnd(names.inheritedFieldLabels)} from this case onto each ${noun}.`);
}
if (names.advanceToName) {
bullets.push(`Moves this case to ${names.advanceToName} as soon as the pieces are created.`);
}
if (config.waitForPieces && names.whenFinishedName) {
bullets.push(`Waits until every ${noun} is finished, then moves this case to ${names.whenFinishedName}.`);
bullets.push(`If the agent returns an empty list, this case skips ahead to ${names.whenFinishedName}.`);
}
return bullets;
}