mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 19:35:13 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The pipeline subsystem models repeatable work as items moving through stages, with agent automation, review gates, blockers, drift notices, and linked work. > - Operators need this to be usable as one coherent workflow surface, not just as backend primitives or disconnected route experiments. > - The branch now carries the pipeline data model, service/routes, CLI/tutorial path, aggregation feeds, operator UI, stage automation controls, liveness/retry handling, and follow-up polish that make the primitive reviewable end to end. > - This pull request is the single review target for that pipeline workflow primitive stack. > - The benefit is that reviewers can evaluate the full operator experience and server contract together against `master`. ## Linked Issues or Issue Description No public GitHub issue exists for this work. The underlying feature request is described inline. ### Problem or motivation Paperclip needs a first-class way to model multi-stage agent/company workflows where upstream items can spawn downstream work, request review, carry fields across pipelines, surface drift, retry automation, and show operators where work is blocked or active. Without a unified pipeline primitive, these workflows spread across ad hoc issues, routines, and comments, making the state hard to inspect or operate. ### Proposed solution Add the pipeline workflow primitive stack: database schema and migrations, shared validators/types, server services and REST routes, aggregation and liveness helpers, CLI/tutorial smoke support, and the React operator UI for pipeline lists, boards, item detail, review/learnings views, settings, stage automation, secrets, carry-over fields, and retry/recovery flows. ### Alternatives considered - Keep workflows as loosely linked issues and routines: rejected because operators need a single board/detail/settings surface for repeated workflow patterns. - Ship backend primitives first and defer UI: rejected for this branch because the operator experience is the main way to validate the primitive. - Add a narrower one-off content workflow: rejected because the same primitives are useful across future company processes. ## What Changed - Added and evolved pipeline schema, migrations, shared contracts, server services, REST routes, route tests, and CLI/tutorial smoke support. - Added pipeline aggregation, health/liveness, drift acknowledgment, blocker/carry-over, automation retry, stage automation environment, and permission recovery behavior. - Added the operator UI for pipeline index/board/item detail/settings/review/learnings flows, including stage secrets, automation controls, markdown/item descriptions, linked issue assets, liveness banners, and source automation metadata. - Refactored issue document frame rendering through the shared `DocumentFrameHeader` component to keep document controls consistent with the pipeline document surfaces. - Kept this PR as the single base-branch review target for the current pipeline branch. ## Verification Current branch refresh: - `pnpm vitest run server/src/__tests__/pipelines-service.test.ts` — 31 passed - `pnpm vitest run server/src/__tests__/pipelines-routes.test.ts` — 19 passed - `pnpm --filter ./server typecheck` — passed - `pnpm --filter ./ui typecheck` — passed - Verified Pipelines remains gated by `enablePipelines === true`: sidebar item is hidden unless the flag is enabled, direct pipeline routes redirect to `/dashboard` when disabled, and the Experimental settings UI still has no Pipelines toggle. - GitHub status checks on `df071c710646de625131064c3fb6588b5e97964a` — all complete with no failing conclusions, including Actions, Socket, Superagent/Security, and Greptile Review - Greptile summary on `df071c710646de625131064c3fb6588b5e97964a` — Confidence Score 5/5 - GitHub review-thread sweep — 0 unresolved Greptile threads Previously recorded during branch development: - Server pipeline service/route and aggregation tests - Shared validator tests - UI pipeline page/settings/item-detail/learnings/liveness tests - Pipeline tutorial smoke path ## Risks - High review surface: this is a large feature branch spanning database, shared contracts, server behavior, CLI/docs, and UI. - Migration ordering and schema compatibility need reviewer attention because this branch has been kept current across multiple `master` syncs. - GitHub still reports merge state `BLOCKED` because the PR is awaiting normal human review/branch-protection completion; all current status checks are green. - Branch-name checklist exception: this PR uses the pre-existing requested branch name, which predates the current public-branch naming rule. The PR title/body avoid internal issue references. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex coding agent based on GPT-5, with repository tool use, shell execution, git/GitHub CLI operations, and local verification commands. Earlier commits in this branch were assisted by Paperclip agents and other AI coding agents as recorded in commit authorship. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
160 lines
7.0 KiB
TypeScript
160 lines
7.0 KiB
TypeScript
import { z } from "zod";
|
|
import {
|
|
ISSUE_EXECUTION_WORKSPACE_PREFERENCES,
|
|
issueExecutionWorkspaceSettingsSchema,
|
|
} from "./issue.js";
|
|
|
|
const routineVariableLikeNameSchema = z.string().trim().regex(/^[A-Za-z][A-Za-z0-9_]*$/);
|
|
|
|
export const pipelineStageKindSchema = z.enum(["working", "review", "done", "cancelled"]);
|
|
export const legacyPipelineStageKindSchema = z.enum(["open", "working", "review", "done", "cancelled"]);
|
|
|
|
export const pipelineStageApproverSchema = z.object({
|
|
kind: z.enum(["any_human", "user", "agent"]).optional().default("any_human"),
|
|
id: z.string().trim().min(1).max(200).optional(),
|
|
}).superRefine((value, ctx) => {
|
|
if (value.kind !== "any_human" && (typeof value.id !== "string" || value.id.length === 0)) {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
path: ["id"],
|
|
message: "Specific stage approvers require an id",
|
|
});
|
|
}
|
|
});
|
|
|
|
export const pipelineStageOnEnterSchema = z.object({
|
|
type: z.literal("run_routine"),
|
|
routineId: z.string().uuid(),
|
|
id: z.string().trim().min(1).max(200).optional(),
|
|
projectId: z.string().uuid().optional().nullable(),
|
|
projectWorkspaceId: z.string().uuid().optional().nullable(),
|
|
executionWorkspaceId: z.string().uuid().optional().nullable(),
|
|
executionWorkspacePreference: z.enum(ISSUE_EXECUTION_WORKSPACE_PREFERENCES).optional().nullable(),
|
|
executionWorkspaceSettings: issueExecutionWorkspaceSettingsSchema.optional().nullable(),
|
|
}).passthrough();
|
|
|
|
export const pipelineStageAutomationSchema = z.object({
|
|
routineId: z.string().uuid().optional().nullable(),
|
|
assigneeAgentId: z.string().uuid().optional().nullable(),
|
|
instructionsBody: z.string().optional().nullable(),
|
|
projectId: z.string().uuid().optional().nullable(),
|
|
projectWorkspaceId: z.string().uuid().optional().nullable(),
|
|
executionWorkspaceId: z.string().uuid().optional().nullable(),
|
|
executionWorkspacePreference: z.enum(ISSUE_EXECUTION_WORKSPACE_PREFERENCES).optional().nullable(),
|
|
executionWorkspaceSettings: issueExecutionWorkspaceSettingsSchema.optional().nullable(),
|
|
}).passthrough();
|
|
|
|
export const pipelineStageCarryOverPolicySchema = z.object({
|
|
version: z.literal(1).default(1),
|
|
mode: z.enum(["all_except", "only"]).default("all_except"),
|
|
includeFields: z.array(routineVariableLikeNameSchema).max(100).default([]),
|
|
excludeFields: z.array(routineVariableLikeNameSchema).max(100).default([]),
|
|
});
|
|
|
|
export const pipelineStageBreakdownSchema = z.object({
|
|
targetPipelineId: z.string().uuid(),
|
|
targetStageKey: z.string().trim().min(1).max(120),
|
|
pieceNoun: z.string().trim().min(1).max(80).default("piece"),
|
|
carryOverPolicy: pipelineStageCarryOverPolicySchema.optional(),
|
|
inheritFields: z.array(routineVariableLikeNameSchema).max(100).default([]),
|
|
advanceTo: z.string().trim().min(1).max(120).optional(),
|
|
waitForPieces: z.boolean().optional().default(false),
|
|
whenFinishedMoveTo: z.string().trim().min(1).max(120).optional(),
|
|
}).superRefine((value, ctx) => {
|
|
if (value.waitForPieces && !value.whenFinishedMoveTo) {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
path: ["whenFinishedMoveTo"],
|
|
message: "Breakdown stages that wait for pieces need a destination stage",
|
|
});
|
|
}
|
|
});
|
|
|
|
export const pipelineStageVariableSchema = z.object({
|
|
key: routineVariableLikeNameSchema,
|
|
label: z.string().trim().max(120),
|
|
type: z.enum(["select", "text", "multiline"]).default("text"),
|
|
options: z.array(z.string().trim().min(1).max(120)).max(50).optional().default([]),
|
|
required: z.boolean().optional().default(false),
|
|
showInAddForm: z.boolean().optional().default(false),
|
|
}).superRefine((value, ctx) => {
|
|
if (value.type === "select" && value.options.length === 0) {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
path: ["options"],
|
|
message: "Select variables require at least one option",
|
|
});
|
|
}
|
|
if (value.type !== "select" && value.options.length > 0) {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
path: ["options"],
|
|
message: "Only select variables can define options",
|
|
});
|
|
}
|
|
});
|
|
|
|
export const pipelineStageConfigSchema = z.object({
|
|
variables: z.array(pipelineStageVariableSchema).default([]),
|
|
disabled: z.boolean().optional(),
|
|
disabledReason: z.string().trim().max(1_000).nullable().optional(),
|
|
requireApproval: z.boolean().optional(),
|
|
approver: pipelineStageApproverSchema.optional(),
|
|
/** Legacy input only; the server migrates it to requireApproval/approver. */
|
|
reviewerKind: z.enum(["human", "any"]).optional(),
|
|
whatHappensHere: z.string().trim().max(10_000).optional(),
|
|
onEnter: pipelineStageOnEnterSchema.optional(),
|
|
automation: pipelineStageAutomationSchema.optional(),
|
|
breakdown: pipelineStageBreakdownSchema.optional(),
|
|
approveToStageKey: z.string().trim().min(1).max(120).optional(),
|
|
rejectToStageKey: z.string().trim().min(1).max(120).optional(),
|
|
requestChangesToStageKey: z.string().trim().min(1).max(120).optional(),
|
|
requireRejectReason: z.boolean().optional(),
|
|
requireRequestChangesReason: z.boolean().optional(),
|
|
requireChildrenTerminal: z.boolean().optional(),
|
|
requireNoUnresolvedDrift: z.boolean().optional(),
|
|
}).passthrough().superRefine((value, ctx) => {
|
|
const keys = new Set<string>();
|
|
value.variables.forEach((variable, index) => {
|
|
if (keys.has(variable.key)) {
|
|
ctx.addIssue({
|
|
code: z.ZodIssueCode.custom,
|
|
path: ["variables", index, "key"],
|
|
message: "Pipeline stage variable keys must be unique",
|
|
});
|
|
}
|
|
keys.add(variable.key);
|
|
});
|
|
});
|
|
|
|
export const pipelineAutomationRetryScopeSchema = z.enum(["current_stage", "previous_stage"]);
|
|
|
|
export const pipelineAutomationRetryCleanupOptionsSchema = z.object({
|
|
retireDirectChildren: z.boolean().default(true),
|
|
retireDescendants: z.boolean().default(true),
|
|
cancelLinkedAutomationIssues: z.boolean().default(true),
|
|
});
|
|
|
|
export const pipelineAutomationRetryRequestSchema = z.object({
|
|
scope: pipelineAutomationRetryScopeSchema,
|
|
targetStageId: z.string().uuid().nullable().optional(),
|
|
expectedVersion: z.number().int().positive(),
|
|
cleanup: pipelineAutomationRetryCleanupOptionsSchema.default({
|
|
retireDirectChildren: true,
|
|
retireDescendants: true,
|
|
cancelLinkedAutomationIssues: true,
|
|
}),
|
|
});
|
|
|
|
export type PipelineStageKind = z.infer<typeof pipelineStageKindSchema>;
|
|
export type PipelineStageApprover = z.infer<typeof pipelineStageApproverSchema>;
|
|
export type PipelineStageOnEnter = z.infer<typeof pipelineStageOnEnterSchema>;
|
|
export type PipelineStageAutomationConfig = z.infer<typeof pipelineStageAutomationSchema>;
|
|
export type PipelineStageCarryOverPolicy = z.infer<typeof pipelineStageCarryOverPolicySchema>;
|
|
export type PipelineStageBreakdown = z.infer<typeof pipelineStageBreakdownSchema>;
|
|
export type PipelineStageVariable = z.infer<typeof pipelineStageVariableSchema>;
|
|
export type PipelineStageConfig = z.infer<typeof pipelineStageConfigSchema>;
|
|
export type PipelineAutomationRetryScope = z.infer<typeof pipelineAutomationRetryScopeSchema>;
|
|
export type PipelineAutomationRetryCleanupOptions = z.infer<typeof pipelineAutomationRetryCleanupOptionsSchema>;
|
|
export type PipelineAutomationRetryRequest = z.infer<typeof pipelineAutomationRetryRequestSchema>;
|