Add pipeline workflow primitives and operator UI (#7903)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The pipeline subsystem models repeatable work as items moving
through stages, with agent automation, review gates, blockers, drift
notices, and linked work.
> - Operators need this to be usable as one coherent workflow surface,
not just as backend primitives or disconnected route experiments.
> - The branch now carries the pipeline data model, service/routes,
CLI/tutorial path, aggregation feeds, operator UI, stage automation
controls, liveness/retry handling, and follow-up polish that make the
primitive reviewable end to end.
> - This pull request is the single review target for that pipeline
workflow primitive stack.
> - The benefit is that reviewers can evaluate the full operator
experience and server contract together against `master`.

## Linked Issues or Issue Description

No public GitHub issue exists for this work. The underlying feature
request is described inline.

### Problem or motivation

Paperclip needs a first-class way to model multi-stage agent/company
workflows where upstream items can spawn downstream work, request
review, carry fields across pipelines, surface drift, retry automation,
and show operators where work is blocked or active. Without a unified
pipeline primitive, these workflows spread across ad hoc issues,
routines, and comments, making the state hard to inspect or operate.

### Proposed solution

Add the pipeline workflow primitive stack: database schema and
migrations, shared validators/types, server services and REST routes,
aggregation and liveness helpers, CLI/tutorial smoke support, and the
React operator UI for pipeline lists, boards, item detail,
review/learnings views, settings, stage automation, secrets, carry-over
fields, and retry/recovery flows.

### Alternatives considered

- Keep workflows as loosely linked issues and routines: rejected because
operators need a single board/detail/settings surface for repeated
workflow patterns.
- Ship backend primitives first and defer UI: rejected for this branch
because the operator experience is the main way to validate the
primitive.
- Add a narrower one-off content workflow: rejected because the same
primitives are useful across future company processes.

## What Changed

- Added and evolved pipeline schema, migrations, shared contracts,
server services, REST routes, route tests, and CLI/tutorial smoke
support.
- Added pipeline aggregation, health/liveness, drift acknowledgment,
blocker/carry-over, automation retry, stage automation environment, and
permission recovery behavior.
- Added the operator UI for pipeline index/board/item
detail/settings/review/learnings flows, including stage secrets,
automation controls, markdown/item descriptions, linked issue assets,
liveness banners, and source automation metadata.
- Refactored issue document frame rendering through the shared
`DocumentFrameHeader` component to keep document controls consistent
with the pipeline document surfaces.
- Kept this PR as the single base-branch review target for the current
pipeline branch.

## Verification

Current branch refresh:

- `pnpm vitest run server/src/__tests__/pipelines-service.test.ts` — 31
passed
- `pnpm vitest run server/src/__tests__/pipelines-routes.test.ts` — 19
passed
- `pnpm --filter ./server typecheck` — passed
- `pnpm --filter ./ui typecheck` — passed
- Verified Pipelines remains gated by `enablePipelines === true`:
sidebar item is hidden unless the flag is enabled, direct pipeline
routes redirect to `/dashboard` when disabled, and the Experimental
settings UI still has no Pipelines toggle.
- GitHub status checks on `df071c710646de625131064c3fb6588b5e97964a` —
all complete with no failing conclusions, including Actions, Socket,
Superagent/Security, and Greptile Review
- Greptile summary on `df071c710646de625131064c3fb6588b5e97964a` —
Confidence Score 5/5
- GitHub review-thread sweep — 0 unresolved Greptile threads

Previously recorded during branch development:

- Server pipeline service/route and aggregation tests
- Shared validator tests
- UI pipeline page/settings/item-detail/learnings/liveness tests
- Pipeline tutorial smoke path

## Risks

- High review surface: this is a large feature branch spanning database,
shared contracts, server behavior, CLI/docs, and UI.
- Migration ordering and schema compatibility need reviewer attention
because this branch has been kept current across multiple `master`
syncs.
- GitHub still reports merge state `BLOCKED` because the PR is awaiting
normal human review/branch-protection completion; all current status
checks are green.
- Branch-name checklist exception: this PR uses the pre-existing
requested branch name, which predates the current public-branch naming
rule. The PR title/body avoid internal issue references.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex coding agent based on GPT-5, with repository tool use,
shell execution, git/GitHub CLI operations, and local verification
commands. Earlier commits in this branch were assisted by Paperclip
agents and other AI coding agents as recorded in commit authorship.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

packages/shared/src/validators/index.ts

@@ -506,6 +506,27 @@ export {
   type CreateAssetImageMetadata,
 } from "./asset.js";
 
+export {
+  pipelineAutomationRetryCleanupOptionsSchema,
+  pipelineAutomationRetryRequestSchema,
+  pipelineAutomationRetryScopeSchema,
+  pipelineStageAutomationSchema,
+  pipelineStageApproverSchema,
+  pipelineStageConfigSchema,
+  pipelineStageKindSchema,
+  pipelineStageOnEnterSchema,
+  pipelineStageVariableSchema,
+  type PipelineAutomationRetryCleanupOptions,
+  type PipelineAutomationRetryRequest,
+  type PipelineAutomationRetryScope,
+  type PipelineStageAutomationConfig,
+  type PipelineStageApprover,
+  type PipelineStageConfig,
+  type PipelineStageKind,
+  type PipelineStageOnEnter,
+  type PipelineStageVariable,
+} from "./pipeline.js";
+
 export {
   createCompanyInviteSchema,
   createOpenClawInvitePromptSchema,

packages/shared/src/validators/instance.ts

@@ -42,6 +42,7 @@ export const instanceExperimentalSettingsSchema = z.object({
   enableEnvironments: z.boolean().default(false),
   enableIsolatedWorkspaces: z.boolean().default(false),
   enableStreamlinedLeftNavigation: z.boolean().default(true),
+  enablePipelines: z.boolean().default(false),
   enableConferenceRoomChat: z.boolean().default(false),
   enableTaskWatchdogs: z.boolean().default(false),
   enableIssuePlanDecompositions: z.boolean().default(false),

packages/shared/src/validators/pipeline.ts

@@ -0,0 +1,159 @@
+import { z } from "zod";
+import {
+  ISSUE_EXECUTION_WORKSPACE_PREFERENCES,
+  issueExecutionWorkspaceSettingsSchema,
+} from "./issue.js";
+
+const routineVariableLikeNameSchema = z.string().trim().regex(/^[A-Za-z][A-Za-z0-9_]*$/);
+
+export const pipelineStageKindSchema = z.enum(["working", "review", "done", "cancelled"]);
+export const legacyPipelineStageKindSchema = z.enum(["open", "working", "review", "done", "cancelled"]);
+
+export const pipelineStageApproverSchema = z.object({
+  kind: z.enum(["any_human", "user", "agent"]).optional().default("any_human"),
+  id: z.string().trim().min(1).max(200).optional(),
+}).superRefine((value, ctx) => {
+  if (value.kind !== "any_human" && (typeof value.id !== "string" || value.id.length === 0)) {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["id"],
+      message: "Specific stage approvers require an id",
+    });
+  }
+});
+
+export const pipelineStageOnEnterSchema = z.object({
+  type: z.literal("run_routine"),
+  routineId: z.string().uuid(),
+  id: z.string().trim().min(1).max(200).optional(),
+  projectId: z.string().uuid().optional().nullable(),
+  projectWorkspaceId: z.string().uuid().optional().nullable(),
+  executionWorkspaceId: z.string().uuid().optional().nullable(),
+  executionWorkspacePreference: z.enum(ISSUE_EXECUTION_WORKSPACE_PREFERENCES).optional().nullable(),
+  executionWorkspaceSettings: issueExecutionWorkspaceSettingsSchema.optional().nullable(),
+}).passthrough();
+
+export const pipelineStageAutomationSchema = z.object({
+  routineId: z.string().uuid().optional().nullable(),
+  assigneeAgentId: z.string().uuid().optional().nullable(),
+  instructionsBody: z.string().optional().nullable(),
+  projectId: z.string().uuid().optional().nullable(),
+  projectWorkspaceId: z.string().uuid().optional().nullable(),
+  executionWorkspaceId: z.string().uuid().optional().nullable(),
+  executionWorkspacePreference: z.enum(ISSUE_EXECUTION_WORKSPACE_PREFERENCES).optional().nullable(),
+  executionWorkspaceSettings: issueExecutionWorkspaceSettingsSchema.optional().nullable(),
+}).passthrough();
+
+export const pipelineStageCarryOverPolicySchema = z.object({
+  version: z.literal(1).default(1),
+  mode: z.enum(["all_except", "only"]).default("all_except"),
+  includeFields: z.array(routineVariableLikeNameSchema).max(100).default([]),
+  excludeFields: z.array(routineVariableLikeNameSchema).max(100).default([]),
+});
+
+export const pipelineStageBreakdownSchema = z.object({
+  targetPipelineId: z.string().uuid(),
+  targetStageKey: z.string().trim().min(1).max(120),
+  pieceNoun: z.string().trim().min(1).max(80).default("piece"),
+  carryOverPolicy: pipelineStageCarryOverPolicySchema.optional(),
+  inheritFields: z.array(routineVariableLikeNameSchema).max(100).default([]),
+  advanceTo: z.string().trim().min(1).max(120).optional(),
+  waitForPieces: z.boolean().optional().default(false),
+  whenFinishedMoveTo: z.string().trim().min(1).max(120).optional(),
+}).superRefine((value, ctx) => {
+  if (value.waitForPieces && !value.whenFinishedMoveTo) {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["whenFinishedMoveTo"],
+      message: "Breakdown stages that wait for pieces need a destination stage",
+    });
+  }
+});
+
+export const pipelineStageVariableSchema = z.object({
+  key: routineVariableLikeNameSchema,
+  label: z.string().trim().max(120),
+  type: z.enum(["select", "text", "multiline"]).default("text"),
+  options: z.array(z.string().trim().min(1).max(120)).max(50).optional().default([]),
+  required: z.boolean().optional().default(false),
+  showInAddForm: z.boolean().optional().default(false),
+}).superRefine((value, ctx) => {
+  if (value.type === "select" && value.options.length === 0) {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["options"],
+      message: "Select variables require at least one option",
+    });
+  }
+  if (value.type !== "select" && value.options.length > 0) {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["options"],
+      message: "Only select variables can define options",
+    });
+  }
+});
+
+export const pipelineStageConfigSchema = z.object({
+  variables: z.array(pipelineStageVariableSchema).default([]),
+  disabled: z.boolean().optional(),
+  disabledReason: z.string().trim().max(1_000).nullable().optional(),
+  requireApproval: z.boolean().optional(),
+  approver: pipelineStageApproverSchema.optional(),
+  /** Legacy input only; the server migrates it to requireApproval/approver. */
+  reviewerKind: z.enum(["human", "any"]).optional(),
+  whatHappensHere: z.string().trim().max(10_000).optional(),
+  onEnter: pipelineStageOnEnterSchema.optional(),
+  automation: pipelineStageAutomationSchema.optional(),
+  breakdown: pipelineStageBreakdownSchema.optional(),
+  approveToStageKey: z.string().trim().min(1).max(120).optional(),
+  rejectToStageKey: z.string().trim().min(1).max(120).optional(),
+  requestChangesToStageKey: z.string().trim().min(1).max(120).optional(),
+  requireRejectReason: z.boolean().optional(),
+  requireRequestChangesReason: z.boolean().optional(),
+  requireChildrenTerminal: z.boolean().optional(),
+  requireNoUnresolvedDrift: z.boolean().optional(),
+}).passthrough().superRefine((value, ctx) => {
+  const keys = new Set<string>();
+  value.variables.forEach((variable, index) => {
+    if (keys.has(variable.key)) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        path: ["variables", index, "key"],
+        message: "Pipeline stage variable keys must be unique",
+      });
+    }
+    keys.add(variable.key);
+  });
+});
+
+export const pipelineAutomationRetryScopeSchema = z.enum(["current_stage", "previous_stage"]);
+
+export const pipelineAutomationRetryCleanupOptionsSchema = z.object({
+  retireDirectChildren: z.boolean().default(true),
+  retireDescendants: z.boolean().default(true),
+  cancelLinkedAutomationIssues: z.boolean().default(true),
+});
+
+export const pipelineAutomationRetryRequestSchema = z.object({
+  scope: pipelineAutomationRetryScopeSchema,
+  targetStageId: z.string().uuid().nullable().optional(),
+  expectedVersion: z.number().int().positive(),
+  cleanup: pipelineAutomationRetryCleanupOptionsSchema.default({
+    retireDirectChildren: true,
+    retireDescendants: true,
+    cancelLinkedAutomationIssues: true,
+  }),
+});
+
+export type PipelineStageKind = z.infer<typeof pipelineStageKindSchema>;
+export type PipelineStageApprover = z.infer<typeof pipelineStageApproverSchema>;
+export type PipelineStageOnEnter = z.infer<typeof pipelineStageOnEnterSchema>;
+export type PipelineStageAutomationConfig = z.infer<typeof pipelineStageAutomationSchema>;
+export type PipelineStageCarryOverPolicy = z.infer<typeof pipelineStageCarryOverPolicySchema>;
+export type PipelineStageBreakdown = z.infer<typeof pipelineStageBreakdownSchema>;
+export type PipelineStageVariable = z.infer<typeof pipelineStageVariableSchema>;
+export type PipelineStageConfig = z.infer<typeof pipelineStageConfigSchema>;
+export type PipelineAutomationRetryScope = z.infer<typeof pipelineAutomationRetryScopeSchema>;
+export type PipelineAutomationRetryCleanupOptions = z.infer<typeof pipelineAutomationRetryCleanupOptionsSchema>;
+export type PipelineAutomationRetryRequest = z.infer<typeof pipelineAutomationRetryRequestSchema>;

packages/shared/src/validators/routine.ts

@@ -159,6 +159,7 @@ export const runRoutineSchema = z.object({
   payload: z.record(z.string(), z.unknown()).optional().nullable(),
   variables: z.record(z.string(), routineVariableValueSchema).optional().nullable(),
   projectId: z.string().uuid().optional().nullable(),
+  projectWorkspaceId: z.string().uuid().optional().nullable(),
   assigneeAgentId: z.string().uuid().optional().nullable(),
   idempotencyKey: z.string().trim().max(255).optional().nullable(),
   source: z.enum(["manual", "api"]).optional().default("manual"),