Files
PaperClipAI/packages/db/src/pipelines-schema.test.ts
T
DottaandPaperclip 43b005b704 Add pipeline workflow primitives and operator UI (#7903)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The pipeline subsystem models repeatable work as items moving
through stages, with agent automation, review gates, blockers, drift
notices, and linked work.
> - Operators need this to be usable as one coherent workflow surface,
not just as backend primitives or disconnected route experiments.
> - The branch now carries the pipeline data model, service/routes,
CLI/tutorial path, aggregation feeds, operator UI, stage automation
controls, liveness/retry handling, and follow-up polish that make the
primitive reviewable end to end.
> - This pull request is the single review target for that pipeline
workflow primitive stack.
> - The benefit is that reviewers can evaluate the full operator
experience and server contract together against `master`.

## Linked Issues or Issue Description

No public GitHub issue exists for this work. The underlying feature
request is described inline.

### Problem or motivation

Paperclip needs a first-class way to model multi-stage agent/company
workflows where upstream items can spawn downstream work, request
review, carry fields across pipelines, surface drift, retry automation,
and show operators where work is blocked or active. Without a unified
pipeline primitive, these workflows spread across ad hoc issues,
routines, and comments, making the state hard to inspect or operate.

### Proposed solution

Add the pipeline workflow primitive stack: database schema and
migrations, shared validators/types, server services and REST routes,
aggregation and liveness helpers, CLI/tutorial smoke support, and the
React operator UI for pipeline lists, boards, item detail,
review/learnings views, settings, stage automation, secrets, carry-over
fields, and retry/recovery flows.

### Alternatives considered

- Keep workflows as loosely linked issues and routines: rejected because
operators need a single board/detail/settings surface for repeated
workflow patterns.
- Ship backend primitives first and defer UI: rejected for this branch
because the operator experience is the main way to validate the
primitive.
- Add a narrower one-off content workflow: rejected because the same
primitives are useful across future company processes.

## What Changed

- Added and evolved pipeline schema, migrations, shared contracts,
server services, REST routes, route tests, and CLI/tutorial smoke
support.
- Added pipeline aggregation, health/liveness, drift acknowledgment,
blocker/carry-over, automation retry, stage automation environment, and
permission recovery behavior.
- Added the operator UI for pipeline index/board/item
detail/settings/review/learnings flows, including stage secrets,
automation controls, markdown/item descriptions, linked issue assets,
liveness banners, and source automation metadata.
- Refactored issue document frame rendering through the shared
`DocumentFrameHeader` component to keep document controls consistent
with the pipeline document surfaces.
- Kept this PR as the single base-branch review target for the current
pipeline branch.

## Verification

Current branch refresh:

- `pnpm vitest run server/src/__tests__/pipelines-service.test.ts` — 31
passed
- `pnpm vitest run server/src/__tests__/pipelines-routes.test.ts` — 19
passed
- `pnpm --filter ./server typecheck` — passed
- `pnpm --filter ./ui typecheck` — passed
- Verified Pipelines remains gated by `enablePipelines === true`:
sidebar item is hidden unless the flag is enabled, direct pipeline
routes redirect to `/dashboard` when disabled, and the Experimental
settings UI still has no Pipelines toggle.
- GitHub status checks on `df071c710646de625131064c3fb6588b5e97964a` —
all complete with no failing conclusions, including Actions, Socket,
Superagent/Security, and Greptile Review
- Greptile summary on `df071c710646de625131064c3fb6588b5e97964a` —
Confidence Score 5/5
- GitHub review-thread sweep — 0 unresolved Greptile threads

Previously recorded during branch development:

- Server pipeline service/route and aggregation tests
- Shared validator tests
- UI pipeline page/settings/item-detail/learnings/liveness tests
- Pipeline tutorial smoke path

## Risks

- High review surface: this is a large feature branch spanning database,
shared contracts, server behavior, CLI/docs, and UI.
- Migration ordering and schema compatibility need reviewer attention
because this branch has been kept current across multiple `master`
syncs.
- GitHub still reports merge state `BLOCKED` because the PR is awaiting
normal human review/branch-protection completion; all current status
checks are green.
- Branch-name checklist exception: this PR uses the pre-existing
requested branch name, which predates the current public-branch naming
rule. The PR title/body avoid internal issue references.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex coding agent based on GPT-5, with repository tool use,
shell execution, git/GitHub CLI operations, and local verification
commands. Earlier commits in this branch were assisted by Paperclip
agents and other AI coding agents as recorded in commit authorship.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-26 12:02:44 -05:00

266 lines
7.6 KiB
TypeScript

import { afterAll, beforeAll, describe, expect, it } from "vitest";
import {
agents,
companies,
createDb,
documents,
issues,
pipelineAutomationExecutions,
pipelineCaseBlockers,
pipelineCaseEvents,
pipelineCaseIssueLinks,
pipelineCases,
pipelineDocuments,
pipelineStages,
pipelineTransitions,
pipelines,
routines,
} from "./index.js";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./test-embedded-postgres.js";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
console.warn(
`Skipping embedded Postgres pipeline schema tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
);
}
function expectConstraintError(action: () => Promise<unknown>) {
return expect(action()).rejects.toThrow("Failed query");
}
describeEmbeddedPostgres("pipeline schema", () => {
let db!: ReturnType<typeof createDb>;
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-pipeline-schema-");
db = createDb(tempDb.connectionString);
}, 20_000);
afterAll(async () => {
await tempDb?.cleanup();
});
it("persists one row per pipeline table and enforces unique keys and required checks", async () => {
const [company] = await db.insert(companies).values({ name: "Pipeline Co", issuePrefix: "PIP" }).returning();
const [agent] = await db.insert(agents).values({
companyId: company.id,
name: "Pipeline Agent",
role: "engineer",
adapterType: "codex_local",
adapterConfig: {},
runtimeConfig: {},
permissions: {},
}).returning();
const [routine] = await db.insert(routines).values({
companyId: company.id,
title: "Draft content",
assigneeAgentId: agent.id,
}).returning();
const [issue] = await db.insert(issues).values({
companyId: company.id,
title: "Work linked to a case",
}).returning();
const [document] = await db.insert(documents).values({
companyId: company.id,
title: "Pipeline guidance",
latestBody: "Use the launch rubric.",
}).returning();
const [pipeline] = await db.insert(pipelines).values({
companyId: company.id,
key: "content",
name: "Content",
createdByAgentId: agent.id,
}).returning();
await expectConstraintError(
() => db.insert(pipelines).values({ companyId: company.id, key: "content", name: "Duplicate" }),
);
const [intakeStage] = await db.insert(pipelineStages).values({
pipelineId: pipeline.id,
key: "intake",
name: "Intake",
kind: "working",
position: 0,
}).returning();
const [reviewStage] = await db.insert(pipelineStages).values({
pipelineId: pipeline.id,
key: "review",
name: "Review",
kind: "review",
position: 1,
}).returning();
await expectConstraintError(
() => db.insert(pipelineStages).values({
pipelineId: pipeline.id,
key: "intake",
name: "Duplicate Intake",
kind: "working",
position: 2,
}),
);
await expectConstraintError(
() => db.insert(pipelineStages).values({
pipelineId: pipeline.id,
key: "invalid",
name: "Invalid",
kind: "waiting",
position: 3,
}),
);
await db.insert(pipelineTransitions).values({
pipelineId: pipeline.id,
fromStageId: intakeStage.id,
toStageId: reviewStage.id,
label: "Send to review",
});
await expectConstraintError(
() => db.insert(pipelineTransitions).values({
pipelineId: pipeline.id,
fromStageId: intakeStage.id,
toStageId: reviewStage.id,
}),
);
const [rootCase] = await db.insert(pipelineCases).values({
companyId: company.id,
pipelineId: pipeline.id,
stageId: intakeStage.id,
caseKey: "release-1",
title: "Release 1",
fields: { channel: "blog" },
workspaceRef: { path: "content/release-1" },
createdByAgentId: agent.id,
}).returning();
const [childCase] = await db.insert(pipelineCases).values({
companyId: company.id,
pipelineId: pipeline.id,
stageId: intakeStage.id,
caseKey: "release-1/blog",
title: "Release 1 blog post",
parentCaseId: rootCase.id,
}).returning();
await expectConstraintError(
() => db.insert(pipelineCases).values({
companyId: company.id,
pipelineId: pipeline.id,
stageId: intakeStage.id,
caseKey: "release-1",
title: "Duplicate case",
}),
);
const [event] = await db.insert(pipelineCaseEvents).values({
companyId: company.id,
caseId: rootCase.id,
type: "ingested",
actorType: "agent",
actorAgentId: agent.id,
runId: "00000000-0000-4000-8000-000000000001",
toStageId: intakeStage.id,
payload: { source: "test" },
}).returning();
await expectConstraintError(
() => db.insert(pipelineCaseEvents).values({
companyId: company.id,
caseId: rootCase.id,
type: "updated",
actorType: "agent",
actorAgentId: agent.id,
}),
);
await db.insert(pipelineCaseIssueLinks).values({
companyId: company.id,
caseId: rootCase.id,
issueId: issue.id,
role: "origin",
createdByRunId: event.runId,
});
await expectConstraintError(
() => db.insert(pipelineCaseIssueLinks).values({
companyId: company.id,
caseId: rootCase.id,
issueId: issue.id,
role: "work",
}),
);
await db.insert(pipelineCaseBlockers).values({
companyId: company.id,
caseId: childCase.id,
blockedByCaseId: rootCase.id,
});
await expectConstraintError(
() => db.insert(pipelineCaseBlockers).values({
companyId: company.id,
caseId: childCase.id,
blockedByCaseId: rootCase.id,
}),
);
await expectConstraintError(
() => db.insert(pipelineCaseBlockers).values({
companyId: company.id,
caseId: rootCase.id,
blockedByCaseId: rootCase.id,
}),
);
await db.insert(pipelineDocuments).values({
companyId: company.id,
pipelineId: pipeline.id,
documentId: document.id,
key: "guidance",
});
await expectConstraintError(
() => db.insert(pipelineDocuments).values({
companyId: company.id,
pipelineId: pipeline.id,
documentId: document.id,
key: "guidance-duplicate-document",
}),
);
const [secondDocument] = await db.insert(documents).values({
companyId: company.id,
title: "Duplicate guidance",
latestBody: "Duplicate key.",
}).returning();
await expectConstraintError(
() => db.insert(pipelineDocuments).values({
companyId: company.id,
pipelineId: pipeline.id,
documentId: secondDocument.id,
key: "guidance",
}),
);
await db.insert(pipelineAutomationExecutions).values({
companyId: company.id,
caseId: rootCase.id,
automationId: "draft-on-enter",
triggeringEventId: event.id,
routineId: routine.id,
status: "succeeded",
executionIssueId: issue.id,
});
await expectConstraintError(
() => db.insert(pipelineAutomationExecutions).values({
companyId: company.id,
caseId: rootCase.id,
automationId: "draft-on-enter",
triggeringEventId: event.id,
routineId: routine.id,
status: "failed",
}),
);
});
});