mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-22 16:51:28 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The pipeline subsystem models repeatable work as items moving through stages, with agent automation, review gates, blockers, drift notices, and linked work. > - Operators need this to be usable as one coherent workflow surface, not just as backend primitives or disconnected route experiments. > - The branch now carries the pipeline data model, service/routes, CLI/tutorial path, aggregation feeds, operator UI, stage automation controls, liveness/retry handling, and follow-up polish that make the primitive reviewable end to end. > - This pull request is the single review target for that pipeline workflow primitive stack. > - The benefit is that reviewers can evaluate the full operator experience and server contract together against `master`. ## Linked Issues or Issue Description No public GitHub issue exists for this work. The underlying feature request is described inline. ### Problem or motivation Paperclip needs a first-class way to model multi-stage agent/company workflows where upstream items can spawn downstream work, request review, carry fields across pipelines, surface drift, retry automation, and show operators where work is blocked or active. Without a unified pipeline primitive, these workflows spread across ad hoc issues, routines, and comments, making the state hard to inspect or operate. ### Proposed solution Add the pipeline workflow primitive stack: database schema and migrations, shared validators/types, server services and REST routes, aggregation and liveness helpers, CLI/tutorial smoke support, and the React operator UI for pipeline lists, boards, item detail, review/learnings views, settings, stage automation, secrets, carry-over fields, and retry/recovery flows. ### Alternatives considered - Keep workflows as loosely linked issues and routines: rejected because operators need a single board/detail/settings surface for repeated workflow patterns. - Ship backend primitives first and defer UI: rejected for this branch because the operator experience is the main way to validate the primitive. - Add a narrower one-off content workflow: rejected because the same primitives are useful across future company processes. ## What Changed - Added and evolved pipeline schema, migrations, shared contracts, server services, REST routes, route tests, and CLI/tutorial smoke support. - Added pipeline aggregation, health/liveness, drift acknowledgment, blocker/carry-over, automation retry, stage automation environment, and permission recovery behavior. - Added the operator UI for pipeline index/board/item detail/settings/review/learnings flows, including stage secrets, automation controls, markdown/item descriptions, linked issue assets, liveness banners, and source automation metadata. - Refactored issue document frame rendering through the shared `DocumentFrameHeader` component to keep document controls consistent with the pipeline document surfaces. - Kept this PR as the single base-branch review target for the current pipeline branch. ## Verification Current branch refresh: - `pnpm vitest run server/src/__tests__/pipelines-service.test.ts` — 31 passed - `pnpm vitest run server/src/__tests__/pipelines-routes.test.ts` — 19 passed - `pnpm --filter ./server typecheck` — passed - `pnpm --filter ./ui typecheck` — passed - Verified Pipelines remains gated by `enablePipelines === true`: sidebar item is hidden unless the flag is enabled, direct pipeline routes redirect to `/dashboard` when disabled, and the Experimental settings UI still has no Pipelines toggle. - GitHub status checks on `df071c710646de625131064c3fb6588b5e97964a` — all complete with no failing conclusions, including Actions, Socket, Superagent/Security, and Greptile Review - Greptile summary on `df071c710646de625131064c3fb6588b5e97964a` — Confidence Score 5/5 - GitHub review-thread sweep — 0 unresolved Greptile threads Previously recorded during branch development: - Server pipeline service/route and aggregation tests - Shared validator tests - UI pipeline page/settings/item-detail/learnings/liveness tests - Pipeline tutorial smoke path ## Risks - High review surface: this is a large feature branch spanning database, shared contracts, server behavior, CLI/docs, and UI. - Migration ordering and schema compatibility need reviewer attention because this branch has been kept current across multiple `master` syncs. - GitHub still reports merge state `BLOCKED` because the PR is awaiting normal human review/branch-protection completion; all current status checks are green. - Branch-name checklist exception: this PR uses the pre-existing requested branch name, which predates the current public-branch naming rule. The PR title/body avoid internal issue references. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex coding agent based on GPT-5, with repository tool use, shell execution, git/GitHub CLI operations, and local verification commands. Earlier commits in this branch were assisted by Paperclip agents and other AI coding agents as recorded in commit authorship. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
266 lines
7.6 KiB
TypeScript
266 lines
7.6 KiB
TypeScript
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
|
import {
|
|
agents,
|
|
companies,
|
|
createDb,
|
|
documents,
|
|
issues,
|
|
pipelineAutomationExecutions,
|
|
pipelineCaseBlockers,
|
|
pipelineCaseEvents,
|
|
pipelineCaseIssueLinks,
|
|
pipelineCases,
|
|
pipelineDocuments,
|
|
pipelineStages,
|
|
pipelineTransitions,
|
|
pipelines,
|
|
routines,
|
|
} from "./index.js";
|
|
import {
|
|
getEmbeddedPostgresTestSupport,
|
|
startEmbeddedPostgresTestDatabase,
|
|
} from "./test-embedded-postgres.js";
|
|
|
|
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
|
|
|
if (!embeddedPostgresSupport.supported) {
|
|
console.warn(
|
|
`Skipping embedded Postgres pipeline schema tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
|
);
|
|
}
|
|
|
|
function expectConstraintError(action: () => Promise<unknown>) {
|
|
return expect(action()).rejects.toThrow("Failed query");
|
|
}
|
|
|
|
describeEmbeddedPostgres("pipeline schema", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
|
|
beforeAll(async () => {
|
|
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-pipeline-schema-");
|
|
db = createDb(tempDb.connectionString);
|
|
}, 20_000);
|
|
|
|
afterAll(async () => {
|
|
await tempDb?.cleanup();
|
|
});
|
|
|
|
it("persists one row per pipeline table and enforces unique keys and required checks", async () => {
|
|
const [company] = await db.insert(companies).values({ name: "Pipeline Co", issuePrefix: "PIP" }).returning();
|
|
const [agent] = await db.insert(agents).values({
|
|
companyId: company.id,
|
|
name: "Pipeline Agent",
|
|
role: "engineer",
|
|
adapterType: "codex_local",
|
|
adapterConfig: {},
|
|
runtimeConfig: {},
|
|
permissions: {},
|
|
}).returning();
|
|
const [routine] = await db.insert(routines).values({
|
|
companyId: company.id,
|
|
title: "Draft content",
|
|
assigneeAgentId: agent.id,
|
|
}).returning();
|
|
const [issue] = await db.insert(issues).values({
|
|
companyId: company.id,
|
|
title: "Work linked to a case",
|
|
}).returning();
|
|
const [document] = await db.insert(documents).values({
|
|
companyId: company.id,
|
|
title: "Pipeline guidance",
|
|
latestBody: "Use the launch rubric.",
|
|
}).returning();
|
|
|
|
const [pipeline] = await db.insert(pipelines).values({
|
|
companyId: company.id,
|
|
key: "content",
|
|
name: "Content",
|
|
createdByAgentId: agent.id,
|
|
}).returning();
|
|
await expectConstraintError(
|
|
() => db.insert(pipelines).values({ companyId: company.id, key: "content", name: "Duplicate" }),
|
|
);
|
|
|
|
const [intakeStage] = await db.insert(pipelineStages).values({
|
|
pipelineId: pipeline.id,
|
|
key: "intake",
|
|
name: "Intake",
|
|
kind: "working",
|
|
position: 0,
|
|
}).returning();
|
|
const [reviewStage] = await db.insert(pipelineStages).values({
|
|
pipelineId: pipeline.id,
|
|
key: "review",
|
|
name: "Review",
|
|
kind: "review",
|
|
position: 1,
|
|
}).returning();
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineStages).values({
|
|
pipelineId: pipeline.id,
|
|
key: "intake",
|
|
name: "Duplicate Intake",
|
|
kind: "working",
|
|
position: 2,
|
|
}),
|
|
);
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineStages).values({
|
|
pipelineId: pipeline.id,
|
|
key: "invalid",
|
|
name: "Invalid",
|
|
kind: "waiting",
|
|
position: 3,
|
|
}),
|
|
);
|
|
|
|
await db.insert(pipelineTransitions).values({
|
|
pipelineId: pipeline.id,
|
|
fromStageId: intakeStage.id,
|
|
toStageId: reviewStage.id,
|
|
label: "Send to review",
|
|
});
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineTransitions).values({
|
|
pipelineId: pipeline.id,
|
|
fromStageId: intakeStage.id,
|
|
toStageId: reviewStage.id,
|
|
}),
|
|
);
|
|
|
|
const [rootCase] = await db.insert(pipelineCases).values({
|
|
companyId: company.id,
|
|
pipelineId: pipeline.id,
|
|
stageId: intakeStage.id,
|
|
caseKey: "release-1",
|
|
title: "Release 1",
|
|
fields: { channel: "blog" },
|
|
workspaceRef: { path: "content/release-1" },
|
|
createdByAgentId: agent.id,
|
|
}).returning();
|
|
const [childCase] = await db.insert(pipelineCases).values({
|
|
companyId: company.id,
|
|
pipelineId: pipeline.id,
|
|
stageId: intakeStage.id,
|
|
caseKey: "release-1/blog",
|
|
title: "Release 1 blog post",
|
|
parentCaseId: rootCase.id,
|
|
}).returning();
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineCases).values({
|
|
companyId: company.id,
|
|
pipelineId: pipeline.id,
|
|
stageId: intakeStage.id,
|
|
caseKey: "release-1",
|
|
title: "Duplicate case",
|
|
}),
|
|
);
|
|
|
|
const [event] = await db.insert(pipelineCaseEvents).values({
|
|
companyId: company.id,
|
|
caseId: rootCase.id,
|
|
type: "ingested",
|
|
actorType: "agent",
|
|
actorAgentId: agent.id,
|
|
runId: "00000000-0000-4000-8000-000000000001",
|
|
toStageId: intakeStage.id,
|
|
payload: { source: "test" },
|
|
}).returning();
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineCaseEvents).values({
|
|
companyId: company.id,
|
|
caseId: rootCase.id,
|
|
type: "updated",
|
|
actorType: "agent",
|
|
actorAgentId: agent.id,
|
|
}),
|
|
);
|
|
|
|
await db.insert(pipelineCaseIssueLinks).values({
|
|
companyId: company.id,
|
|
caseId: rootCase.id,
|
|
issueId: issue.id,
|
|
role: "origin",
|
|
createdByRunId: event.runId,
|
|
});
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineCaseIssueLinks).values({
|
|
companyId: company.id,
|
|
caseId: rootCase.id,
|
|
issueId: issue.id,
|
|
role: "work",
|
|
}),
|
|
);
|
|
|
|
await db.insert(pipelineCaseBlockers).values({
|
|
companyId: company.id,
|
|
caseId: childCase.id,
|
|
blockedByCaseId: rootCase.id,
|
|
});
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineCaseBlockers).values({
|
|
companyId: company.id,
|
|
caseId: childCase.id,
|
|
blockedByCaseId: rootCase.id,
|
|
}),
|
|
);
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineCaseBlockers).values({
|
|
companyId: company.id,
|
|
caseId: rootCase.id,
|
|
blockedByCaseId: rootCase.id,
|
|
}),
|
|
);
|
|
|
|
await db.insert(pipelineDocuments).values({
|
|
companyId: company.id,
|
|
pipelineId: pipeline.id,
|
|
documentId: document.id,
|
|
key: "guidance",
|
|
});
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineDocuments).values({
|
|
companyId: company.id,
|
|
pipelineId: pipeline.id,
|
|
documentId: document.id,
|
|
key: "guidance-duplicate-document",
|
|
}),
|
|
);
|
|
const [secondDocument] = await db.insert(documents).values({
|
|
companyId: company.id,
|
|
title: "Duplicate guidance",
|
|
latestBody: "Duplicate key.",
|
|
}).returning();
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineDocuments).values({
|
|
companyId: company.id,
|
|
pipelineId: pipeline.id,
|
|
documentId: secondDocument.id,
|
|
key: "guidance",
|
|
}),
|
|
);
|
|
|
|
await db.insert(pipelineAutomationExecutions).values({
|
|
companyId: company.id,
|
|
caseId: rootCase.id,
|
|
automationId: "draft-on-enter",
|
|
triggeringEventId: event.id,
|
|
routineId: routine.id,
|
|
status: "succeeded",
|
|
executionIssueId: issue.id,
|
|
});
|
|
await expectConstraintError(
|
|
() => db.insert(pipelineAutomationExecutions).values({
|
|
companyId: company.id,
|
|
caseId: rootCase.id,
|
|
automationId: "draft-on-enter",
|
|
triggeringEventId: event.id,
|
|
routineId: routine.id,
|
|
status: "failed",
|
|
}),
|
|
);
|
|
});
|
|
});
|