Files
PaperClipAI/server/src/__tests__/openapi-routes.test.ts
T
DottaandPaperclip 43b005b704 Add pipeline workflow primitives and operator UI (#7903)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The pipeline subsystem models repeatable work as items moving
through stages, with agent automation, review gates, blockers, drift
notices, and linked work.
> - Operators need this to be usable as one coherent workflow surface,
not just as backend primitives or disconnected route experiments.
> - The branch now carries the pipeline data model, service/routes,
CLI/tutorial path, aggregation feeds, operator UI, stage automation
controls, liveness/retry handling, and follow-up polish that make the
primitive reviewable end to end.
> - This pull request is the single review target for that pipeline
workflow primitive stack.
> - The benefit is that reviewers can evaluate the full operator
experience and server contract together against `master`.

## Linked Issues or Issue Description

No public GitHub issue exists for this work. The underlying feature
request is described inline.

### Problem or motivation

Paperclip needs a first-class way to model multi-stage agent/company
workflows where upstream items can spawn downstream work, request
review, carry fields across pipelines, surface drift, retry automation,
and show operators where work is blocked or active. Without a unified
pipeline primitive, these workflows spread across ad hoc issues,
routines, and comments, making the state hard to inspect or operate.

### Proposed solution

Add the pipeline workflow primitive stack: database schema and
migrations, shared validators/types, server services and REST routes,
aggregation and liveness helpers, CLI/tutorial smoke support, and the
React operator UI for pipeline lists, boards, item detail,
review/learnings views, settings, stage automation, secrets, carry-over
fields, and retry/recovery flows.

### Alternatives considered

- Keep workflows as loosely linked issues and routines: rejected because
operators need a single board/detail/settings surface for repeated
workflow patterns.
- Ship backend primitives first and defer UI: rejected for this branch
because the operator experience is the main way to validate the
primitive.
- Add a narrower one-off content workflow: rejected because the same
primitives are useful across future company processes.

## What Changed

- Added and evolved pipeline schema, migrations, shared contracts,
server services, REST routes, route tests, and CLI/tutorial smoke
support.
- Added pipeline aggregation, health/liveness, drift acknowledgment,
blocker/carry-over, automation retry, stage automation environment, and
permission recovery behavior.
- Added the operator UI for pipeline index/board/item
detail/settings/review/learnings flows, including stage secrets,
automation controls, markdown/item descriptions, linked issue assets,
liveness banners, and source automation metadata.
- Refactored issue document frame rendering through the shared
`DocumentFrameHeader` component to keep document controls consistent
with the pipeline document surfaces.
- Kept this PR as the single base-branch review target for the current
pipeline branch.

## Verification

Current branch refresh:

- `pnpm vitest run server/src/__tests__/pipelines-service.test.ts` — 31
passed
- `pnpm vitest run server/src/__tests__/pipelines-routes.test.ts` — 19
passed
- `pnpm --filter ./server typecheck` — passed
- `pnpm --filter ./ui typecheck` — passed
- Verified Pipelines remains gated by `enablePipelines === true`:
sidebar item is hidden unless the flag is enabled, direct pipeline
routes redirect to `/dashboard` when disabled, and the Experimental
settings UI still has no Pipelines toggle.
- GitHub status checks on `df071c710646de625131064c3fb6588b5e97964a` —
all complete with no failing conclusions, including Actions, Socket,
Superagent/Security, and Greptile Review
- Greptile summary on `df071c710646de625131064c3fb6588b5e97964a` —
Confidence Score 5/5
- GitHub review-thread sweep — 0 unresolved Greptile threads

Previously recorded during branch development:

- Server pipeline service/route and aggregation tests
- Shared validator tests
- UI pipeline page/settings/item-detail/learnings/liveness tests
- Pipeline tutorial smoke path

## Risks

- High review surface: this is a large feature branch spanning database,
shared contracts, server behavior, CLI/docs, and UI.
- Migration ordering and schema compatibility need reviewer attention
because this branch has been kept current across multiple `master`
syncs.
- GitHub still reports merge state `BLOCKED` because the PR is awaiting
normal human review/branch-protection completion; all current status
checks are green.
- Branch-name checklist exception: this PR uses the pre-existing
requested branch name, which predates the current public-branch naming
rule. The PR title/body avoid internal issue references.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex coding agent based on GPT-5, with repository tool use,
shell execution, git/GitHub CLI operations, and local verification
commands. Earlier commits in this branch were assisted by Paperclip
agents and other AI coding agents as recorded in commit authorship.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-26 12:02:44 -05:00

196 lines
6.9 KiB
TypeScript

import express from "express";
import fs from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import request from "supertest";
import { describe, expect, it } from "vitest";
import { errorHandler } from "../middleware/index.js";
import { buildOpenApiSpec, openApiRoutes } from "../routes/openapi.js";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const ROUTES_DIR = path.resolve(__dirname, "../routes");
const apiPrefixes: Record<string, string> = {
"access.ts": "/api",
"activity.ts": "/api",
"adapters.ts": "/api",
"agents.ts": "/api",
"approvals.ts": "/api",
"assets.ts": "/api",
"auth.ts": "/api/auth",
"board-chat.ts": "/api",
"cloud-upstreams.ts": "/api",
"companies.ts": "/api/companies",
"company-skills.ts": "/api",
"costs.ts": "/api",
"dashboard.ts": "/api",
"environments.ts": "/api",
"execution-workspaces.ts": "/api",
"file-resources.ts": "/api",
"goals.ts": "/api",
"health.ts": "/api/health",
"inbox-dismissals.ts": "/api",
"instance-database-backups.ts": "/api",
"instance-settings.ts": "/api",
"issues.ts": "/api",
"issue-tree-control.ts": "/api",
"llms.ts": "/api",
"openapi.ts": "/api",
"plugin-ui-static.ts": "/api",
"plugins.ts": "/api",
"projects.ts": "/api",
"resource-memberships.ts": "/api",
"routines.ts": "/api",
"secrets.ts": "/api",
"sidebar-badges.ts": "/api",
"sidebar-preferences.ts": "/api",
"teams-catalog.ts": "/api",
"user-profiles.ts": "/api",
};
const ROUTE_LITERAL_PATTERN = /router\.(get|post|put|patch|delete)\(\s*["'`]([^"'`]+)["'`]/g;
const ROUTER_METHOD_PATTERN = /router\.(get|post|put|patch|delete)\(/;
const HTTP_METHODS = new Set(["get", "put", "post", "delete", "options", "head", "patch", "trace"]);
const explicitOpenApiCoverageExclusions = new Set([
// Pipeline routes are experimental and not yet represented in the public OpenAPI document.
"pipelines.ts",
]);
function createApp() {
const app = express();
app.use("/api", openApiRoutes());
app.use(errorHandler);
return app;
}
function normalizeExpressPath(routePath: string) {
return routePath
.replace(/\*([A-Za-z0-9_]+)/g, "{$1}")
.replace(/:([A-Za-z0-9_]+)/g, "{$1}")
.replace(/\/+/g, "/");
}
function resolveMountedPath(file: string, prefix: string, routePath: string) {
if ((file === "companies.ts" || file === "health.ts") && routePath === "/") {
return prefix;
}
if (file === "companies.ts" || file === "health.ts") {
return `${prefix}${routePath}`;
}
if (file === "auth.ts") {
return `${prefix}${routePath === "/" ? "" : routePath}`;
}
return `${prefix}${routePath}`;
}
function loadActualRoutes() {
const routes = new Set<string>();
const unknownRouteFiles: string[] = [];
for (const file of fs.readdirSync(ROUTES_DIR).filter((entry) => entry.endsWith(".ts"))) {
if (explicitOpenApiCoverageExclusions.has(file)) continue;
const prefix = apiPrefixes[file];
const source = fs.readFileSync(path.join(ROUTES_DIR, file), "utf8");
if (!prefix) {
if (ROUTER_METHOD_PATTERN.test(source)) {
unknownRouteFiles.push(file);
}
continue;
}
for (const match of source.matchAll(ROUTE_LITERAL_PATTERN)) {
const method = match[1].toUpperCase();
const routePath = match[2];
routes.add(`${method} ${normalizeExpressPath(resolveMountedPath(file, prefix, routePath))}`);
}
if (file === "companies.ts" && source.includes("router.post(COMPANY_IMPORT_ROUTE_PATH")) {
routes.add("POST /api/companies/import");
}
}
return { routes, unknownRouteFiles: unknownRouteFiles.sort() };
}
function loadSpecRoutes() {
const spec = buildOpenApiSpec();
const routes = new Set<string>();
for (const [routePath, pathItem] of Object.entries<Record<string, Record<string, unknown>>>(spec.paths ?? {})) {
for (const method of Object.keys(pathItem)) {
if (HTTP_METHODS.has(method)) {
routes.add(`${method.toUpperCase()} ${routePath}`);
}
}
}
return { spec, routes };
}
describe("openapi routes", () => {
it("serves the generated OpenAPI document", async () => {
const res = await request(createApp()).get("/api/openapi.json");
expect(res.status).toBe(200);
expect(res.body.openapi).toBe("3.0.0");
expect(res.body.info.title).toBe("Paperclip API");
expect(res.body.paths["/api/openapi.json"].get.summary).toBe("Get the generated OpenAPI document");
expect(res.body.paths["/api/companies/{companyId}/agents"].get.summary).toBe("List agents in a company");
expect(res.body.paths["/api/agents/{id}/keys"].post.summary).toBe("Create an agent API key");
expect(res.body.components.securitySchemes).toMatchObject({
BoardSessionAuth: { type: "apiKey", in: "cookie" },
BoardApiKeyAuth: { type: "http", scheme: "bearer" },
AgentBearerAuth: { type: "http", scheme: "bearer" },
});
expect(res.body.paths["/api/health"].get.security).toEqual([]);
expect(res.body.paths["/api/companies"].post.responses["201"]).toBeDefined();
expect(res.body.paths["/api/companies"].post.requestBody.content["application/json"].schema).toMatchObject({
type: "object",
properties: {
name: { type: "string", minLength: 1 },
},
required: ["name"],
});
expect(res.body.paths["/api/agents/{id}/keys"].post.requestBody.content["application/json"].schema).toMatchObject({
type: "object",
properties: {
name: { type: "string" },
},
});
});
it("covers the mounted server routes exactly", () => {
const { routes: actualRoutes, unknownRouteFiles } = loadActualRoutes();
const { routes: specRoutes } = loadSpecRoutes();
const missingInSpec = [...actualRoutes].filter((route) => !specRoutes.has(route)).sort();
const extraInSpec = [...specRoutes].filter((route) => !actualRoutes.has(route)).sort();
expect({ unknownRouteFiles, missingInSpec, extraInSpec }).toEqual({
unknownRouteFiles: [],
missingInSpec: [],
extraInSpec: [],
});
});
it("documents auth and reviewed response-code invariants", () => {
const { spec } = loadSpecRoutes();
expect(spec.paths["/api/openapi.json"].get.security).toEqual([]);
expect(spec.paths["/api/plugins/install"].post.security).toEqual([
{ BoardSessionAuth: [] },
{ BoardApiKeyAuth: [] },
]);
expect(spec.paths["/api/plugins/install"].post["x-paperclip-authorization"]).toEqual({
actor: "board",
instanceAdmin: true,
});
expect(spec.paths["/api/companies/{companyId}/cost-events"].post.responses["201"]).toBeDefined();
expect(spec.paths["/api/companies/{companyId}/cost-events"].post.responses["403"]).toBeDefined();
expect(spec.paths["/api/instance/database-backups"].post.responses["201"]).toBeDefined();
expect(spec.paths["/api/invites/{token}/accept"].post.responses["202"]).toBeDefined();
expect(spec.paths["/api/board-api-keys"].post.responses["201"]).toBeDefined();
expect(spec.paths["/api/companies/import"].post.responses["202"]).toBeDefined();
});
});