mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-19 20:44:16 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The pipeline subsystem models repeatable work as items moving through stages, with agent automation, review gates, blockers, drift notices, and linked work. > - Operators need this to be usable as one coherent workflow surface, not just as backend primitives or disconnected route experiments. > - The branch now carries the pipeline data model, service/routes, CLI/tutorial path, aggregation feeds, operator UI, stage automation controls, liveness/retry handling, and follow-up polish that make the primitive reviewable end to end. > - This pull request is the single review target for that pipeline workflow primitive stack. > - The benefit is that reviewers can evaluate the full operator experience and server contract together against `master`. ## Linked Issues or Issue Description No public GitHub issue exists for this work. The underlying feature request is described inline. ### Problem or motivation Paperclip needs a first-class way to model multi-stage agent/company workflows where upstream items can spawn downstream work, request review, carry fields across pipelines, surface drift, retry automation, and show operators where work is blocked or active. Without a unified pipeline primitive, these workflows spread across ad hoc issues, routines, and comments, making the state hard to inspect or operate. ### Proposed solution Add the pipeline workflow primitive stack: database schema and migrations, shared validators/types, server services and REST routes, aggregation and liveness helpers, CLI/tutorial smoke support, and the React operator UI for pipeline lists, boards, item detail, review/learnings views, settings, stage automation, secrets, carry-over fields, and retry/recovery flows. ### Alternatives considered - Keep workflows as loosely linked issues and routines: rejected because operators need a single board/detail/settings surface for repeated workflow patterns. - Ship backend primitives first and defer UI: rejected for this branch because the operator experience is the main way to validate the primitive. - Add a narrower one-off content workflow: rejected because the same primitives are useful across future company processes. ## What Changed - Added and evolved pipeline schema, migrations, shared contracts, server services, REST routes, route tests, and CLI/tutorial smoke support. - Added pipeline aggregation, health/liveness, drift acknowledgment, blocker/carry-over, automation retry, stage automation environment, and permission recovery behavior. - Added the operator UI for pipeline index/board/item detail/settings/review/learnings flows, including stage secrets, automation controls, markdown/item descriptions, linked issue assets, liveness banners, and source automation metadata. - Refactored issue document frame rendering through the shared `DocumentFrameHeader` component to keep document controls consistent with the pipeline document surfaces. - Kept this PR as the single base-branch review target for the current pipeline branch. ## Verification Current branch refresh: - `pnpm vitest run server/src/__tests__/pipelines-service.test.ts` — 31 passed - `pnpm vitest run server/src/__tests__/pipelines-routes.test.ts` — 19 passed - `pnpm --filter ./server typecheck` — passed - `pnpm --filter ./ui typecheck` — passed - Verified Pipelines remains gated by `enablePipelines === true`: sidebar item is hidden unless the flag is enabled, direct pipeline routes redirect to `/dashboard` when disabled, and the Experimental settings UI still has no Pipelines toggle. - GitHub status checks on `df071c710646de625131064c3fb6588b5e97964a` — all complete with no failing conclusions, including Actions, Socket, Superagent/Security, and Greptile Review - Greptile summary on `df071c710646de625131064c3fb6588b5e97964a` — Confidence Score 5/5 - GitHub review-thread sweep — 0 unresolved Greptile threads Previously recorded during branch development: - Server pipeline service/route and aggregation tests - Shared validator tests - UI pipeline page/settings/item-detail/learnings/liveness tests - Pipeline tutorial smoke path ## Risks - High review surface: this is a large feature branch spanning database, shared contracts, server behavior, CLI/docs, and UI. - Migration ordering and schema compatibility need reviewer attention because this branch has been kept current across multiple `master` syncs. - GitHub still reports merge state `BLOCKED` because the PR is awaiting normal human review/branch-protection completion; all current status checks are green. - Branch-name checklist exception: this PR uses the pre-existing requested branch name, which predates the current public-branch naming rule. The PR title/body avoid internal issue references. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex coding agent based on GPT-5, with repository tool use, shell execution, git/GitHub CLI operations, and local verification commands. Earlier commits in this branch were assisted by Paperclip agents and other AI coding agents as recorded in commit authorship. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
144 lines
5.1 KiB
TypeScript
144 lines
5.1 KiB
TypeScript
/**
|
|
* Typed work references for pipeline cases.
|
|
*
|
|
* Cases carry references to the actual work — a workspace folder, an external
|
|
* URL, a linked issue — inside `fields` (and the dedicated `workspaceRef`
|
|
* column). This module formalises those loosely-shaped values into a small
|
|
* typed union so the case detail panel can render real links/chips instead of
|
|
* dumping "[object Object]" or "Added details" into the plain field list.
|
|
*
|
|
* Detection is intentionally tolerant: references can arrive as explicit
|
|
* `{ kind: "url", url }` records, as bare URL strings, or as records that simply
|
|
* carry a tell-tale field (`url`, `issueId`, `path`). Anything we don't
|
|
* recognise is left in the plain Details list untouched.
|
|
*/
|
|
|
|
export type WorkReference =
|
|
| { id: string; kind: "workspace"; label: string; path: string | null; branch: string | null }
|
|
| { id: string; kind: "url"; label: string; url: string }
|
|
| { id: string; kind: "issue"; label: string; issueId: string | null; identifier: string | null };
|
|
|
|
interface ReferenceCaseInput {
|
|
fields?: Record<string, unknown> | null;
|
|
workspaceRef?: Record<string, unknown> | null;
|
|
}
|
|
|
|
function readString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim() ? value.trim() : null;
|
|
}
|
|
|
|
function readRecord(value: unknown): Record<string, unknown> | null {
|
|
return value && typeof value === "object" && !Array.isArray(value) ? (value as Record<string, unknown>) : null;
|
|
}
|
|
|
|
function isHttpUrl(value: string): boolean {
|
|
return /^https?:\/\//i.test(value.trim());
|
|
}
|
|
|
|
function humanizeKey(key: string): string {
|
|
return key
|
|
.replace(/[_-]+/g, " ")
|
|
.replace(/([a-z0-9])([A-Z])/g, "$1 $2")
|
|
.trim()
|
|
.replace(/\s+/g, " ")
|
|
.replace(/^./, (char) => char.toUpperCase());
|
|
}
|
|
|
|
function normalizeKind(raw: unknown): WorkReference["kind"] | null {
|
|
const kind = readString(raw)?.toLowerCase();
|
|
if (!kind) return null;
|
|
if (kind === "workspace" || kind === "folder" || kind === "workspace_folder" || kind === "file") return "workspace";
|
|
if (kind === "url" || kind === "link" || kind === "external") return "url";
|
|
if (kind === "issue" || kind === "task" || kind === "ticket") return "issue";
|
|
return null;
|
|
}
|
|
|
|
function workspaceFromRecord(id: string, label: string, record: Record<string, unknown>): WorkReference {
|
|
return {
|
|
id,
|
|
kind: "workspace",
|
|
label,
|
|
path:
|
|
readString(record.path) ??
|
|
readString(record.folder) ??
|
|
readString(record.workspacePath) ??
|
|
readString(record.name) ??
|
|
readString(record.label),
|
|
branch: readString(record.branch) ?? readString(record.ref),
|
|
};
|
|
}
|
|
|
|
/** Parse a single `fields` entry into a typed reference, or null if it isn't one. */
|
|
function referenceFromField(key: string, value: unknown): WorkReference | null {
|
|
const label = humanizeKey(key);
|
|
|
|
const url = readString(value);
|
|
if (url && isHttpUrl(url)) {
|
|
return { id: key, kind: "url", label, url };
|
|
}
|
|
|
|
const record = readRecord(value);
|
|
if (!record) return null;
|
|
|
|
const explicitKind = normalizeKind(record.kind ?? record.type);
|
|
|
|
// URL-shaped.
|
|
const recordUrl = readString(record.url) ?? readString(record.href);
|
|
if (explicitKind === "url" || (recordUrl && isHttpUrl(recordUrl))) {
|
|
if (!recordUrl) return null;
|
|
return { id: key, kind: "url", label: readString(record.label) ?? label, url: recordUrl };
|
|
}
|
|
|
|
// Issue-shaped.
|
|
const issueId = readString(record.issueId) ?? readString(record.id);
|
|
const identifier = readString(record.identifier) ?? readString(record.issueIdentifier);
|
|
if (explicitKind === "issue" || ((issueId || identifier) && (record.issueId || record.issueIdentifier))) {
|
|
return {
|
|
id: key,
|
|
kind: "issue",
|
|
label: readString(record.title) ?? readString(record.label) ?? label,
|
|
issueId,
|
|
identifier,
|
|
};
|
|
}
|
|
|
|
// Workspace-shaped.
|
|
if (explicitKind === "workspace" || record.path || record.folder || record.workspacePath) {
|
|
return workspaceFromRecord(key, readString(record.label) ?? label, record);
|
|
}
|
|
|
|
return null;
|
|
}
|
|
|
|
/**
|
|
* Extract every typed work reference for a case, starting with the dedicated
|
|
* `workspaceRef` column and then any reference-shaped `fields` entries.
|
|
*/
|
|
export function extractWorkReferences(caseItem: ReferenceCaseInput): WorkReference[] {
|
|
const references: WorkReference[] = [];
|
|
|
|
const workspaceRef = readRecord(caseItem.workspaceRef);
|
|
if (workspaceRef && (workspaceRef.path || workspaceRef.folder || workspaceRef.workspacePath || workspaceRef.name)) {
|
|
references.push(workspaceFromRecord("workspaceRef", "Workspace folder", workspaceRef));
|
|
}
|
|
|
|
for (const [key, value] of Object.entries(caseItem.fields ?? {})) {
|
|
const reference = referenceFromField(key, value);
|
|
if (reference) references.push(reference);
|
|
}
|
|
|
|
return references;
|
|
}
|
|
|
|
/**
|
|
* The set of `fields` keys that render as typed references, so the plain
|
|
* Details list can exclude them and avoid showing the same value twice.
|
|
*/
|
|
export function referenceFieldKeys(fields: Record<string, unknown> | null | undefined): Set<string> {
|
|
const keys = new Set<string>();
|
|
for (const [key, value] of Object.entries(fields ?? {})) {
|
|
if (referenceFromField(key, value)) keys.add(key);
|
|
}
|
|
return keys;
|
|
}
|