mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 15:35:12 +01:00
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source control plane for companies of AI agents. > - Operators use the attention feed to find decisions that need action. > - The feed has eleven source kinds, but it has no durable queue or triage state. > - The feed also returns every item and lacks decision deadlines, snooze state, and decision-focused ordering. > - This pull request adds secure queue sidecars and enriches the attention feed with triage data, filters, cursor pagination, and decide-now ranking. > - The benefit is a bounded feed that can show the most urgent decisions first without weakening source visibility rules. ## Linked Issues or Issue Description This pull request replaces the closed [#10634](https://github.com/paperclipai/paperclip/pull/10634). It combines that queue foundation with the dependent attention-feed change as one review unit. **Subsystem affected** Database schema, shared contracts, server authorization and REST APIs, and the UI attention client library. **Problem or motivation** The attention feed can contain hundreds of mixed decision items. Operators cannot group them into durable queues, set a decision deadline, snooze an item, or request a bounded page ordered by urgency. The current client must download the full feed on each refresh. **Proposed solution** Store queue membership and triage state by stable attention identity. Re-authorize each source during queue reads and writes. Enrich attention items with queue, deadline, snooze, expiry, rule, and origin data. Add activity and queue filters, opaque cursor pagination, decide-focused ordering, and a decide-now count. **Alternatives considered** Adding queue fields to every source would duplicate schema and authorization logic across eleven source kinds. Client-only filtering and sorting would still transfer the full feed and would make pagination unstable. **Roadmap alignment** This change improves the core decision-attention surface and operator oversight. It does not implement the separate general-purpose work queue milestone in `ROADMAP.md`. ## What Changed - Added company-scoped queue, membership, triage, and append-only event tables with actor and run provenance. - Added queue CRUD, item membership, starter-rule discovery, and decide-by and snooze endpoints. - Kept source authorization on each queue mutation, read, and count. - Added attention fields for expiry, rule, origin agent, queues, decide-by attribution, and snooze state. - Added activity date filters, queue filters, opaque cursor pagination, and configurable page limits. - Added decide-now ordering by deadline, expiry, severity, and activity. - Added `decideNowCount` and excluded actively snoozed items from the default feed. - Updated the shared and UI client contracts. - Added focused server, route, OpenAPI, and UI client tests. ## Verification - `pnpm exec vitest run server/src/__tests__/attention-service.test.ts server/src/__tests__/decision-queues-routes.test.ts server/src/__tests__/openapi-routes.test.ts ui/src/api/attention.test.ts ui/src/lib/attention.test.ts` (72 tests passed) - `pnpm --filter @paperclipai/db check:migrations` - `pnpm -r --filter @paperclipai/db --filter @paperclipai/shared --filter @paperclipai/server --filter @paperclipai/ui typecheck` - `git diff --check origin/master...HEAD` ## Risks - The migration adds four company-scoped tables and provenance foreign keys. Migration numbering and safety checks pass. - Attention reads can lazily create starter queues and memberships. Inserts are idempotent, audited, and transactional. - Cursor validity depends on the filtered feed. The API returns a clear validation error when the cursor item no longer exists in that feed. - Queue reads re-check source visibility. This favors correct authorization over fewer queries. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, model `gpt-5`. The runtime used agentic reasoning, repository tools, code execution, and test execution. The runtime did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
259 lines
9.9 KiB
TypeScript
259 lines
9.9 KiB
TypeScript
import express from "express";
|
|
import fs from "node:fs";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
import request from "supertest";
|
|
import { describe, expect, it } from "vitest";
|
|
import { errorHandler } from "../middleware/index.js";
|
|
import { buildOpenApiSpec, openApiRoutes } from "../routes/openapi.js";
|
|
|
|
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
|
const ROUTES_DIR = path.resolve(__dirname, "../routes");
|
|
|
|
const apiPrefixes: Record<string, string> = {
|
|
"access.ts": "/api",
|
|
"activity.ts": "/api",
|
|
"adapters.ts": "/api",
|
|
"agents.ts": "/api",
|
|
"attention.ts": "/api",
|
|
"approvals.ts": "/api",
|
|
"assets.ts": "/api",
|
|
"auth.ts": "/api/auth",
|
|
"board-chat.ts": "/api",
|
|
"built-in-agents.ts": "/api",
|
|
"companies.ts": "/api/companies",
|
|
"company-skills.ts": "/api",
|
|
"company-skill-policy.ts": "/api",
|
|
"costs.ts": "/api",
|
|
"dashboard.ts": "/api",
|
|
"decision-queues.ts": "/api",
|
|
"decisions.ts": "/api",
|
|
"decision-training.ts": "/api",
|
|
"environments.ts": "/api",
|
|
"execution-workspaces.ts": "/api",
|
|
"file-resources.ts": "/api",
|
|
"folders.ts": "/api",
|
|
"goals.ts": "/api",
|
|
"health.ts": "/api/health",
|
|
"inbox-agent-policy.ts": "/api",
|
|
"inbox-dismissals.ts": "/api",
|
|
"instance-database-backups.ts": "/api",
|
|
"instance-settings.ts": "/api",
|
|
"issues.ts": "/api",
|
|
"issue-tree-control.ts": "/api",
|
|
"llms.ts": "/api",
|
|
"openapi.ts": "/api",
|
|
"plugin-ui-static.ts": "/api",
|
|
"plugins.ts": "/api",
|
|
"projects.ts": "/api",
|
|
"resource-memberships.ts": "/api",
|
|
"routines.ts": "/api",
|
|
"secrets.ts": "/api",
|
|
"sidebar-badges.ts": "/api",
|
|
"sidebar-preferences.ts": "/api",
|
|
"summary-slots.ts": "/api",
|
|
"status-cards.ts": "/api",
|
|
"teams-catalog.ts": "/api",
|
|
"tool-access.ts": "/api",
|
|
"tool-gateway.ts": "/api",
|
|
"user-profiles.ts": "/api",
|
|
};
|
|
|
|
const ROUTE_LITERAL_PATTERN = /router\.(get|post|put|patch|delete)\(\s*["'`]([^"'`]+)["'`]/g;
|
|
const ROUTER_METHOD_PATTERN = /router\.(get|post|put|patch|delete)\(/;
|
|
const HTTP_METHODS = new Set(["get", "put", "post", "delete", "options", "head", "patch", "trace"]);
|
|
const explicitOpenApiCoverageExclusions = new Set([
|
|
// Pipeline routes are experimental and not yet represented in the public OpenAPI document.
|
|
"pipelines.ts",
|
|
// Case routes are experimental (enableCases flag) and not yet in the public OpenAPI document.
|
|
"cases.ts",
|
|
// Smoke lab routes are experimental and not yet represented in the public OpenAPI document.
|
|
"smoke-lab.ts",
|
|
]);
|
|
|
|
function createApp() {
|
|
const app = express();
|
|
app.use("/api", openApiRoutes());
|
|
app.use(errorHandler);
|
|
return app;
|
|
}
|
|
|
|
function normalizeExpressPath(routePath: string) {
|
|
return routePath
|
|
.replace(/\*([A-Za-z0-9_]+)/g, "{$1}")
|
|
.replace(/:([A-Za-z0-9_]+)/g, "{$1}")
|
|
.replace(/\/+/g, "/");
|
|
}
|
|
|
|
function resolveMountedPath(file: string, prefix: string, routePath: string) {
|
|
if (file === "tool-gateway.ts" && routePath.startsWith("/mcp/gateways/")) {
|
|
return routePath;
|
|
}
|
|
if ((file === "companies.ts" || file === "health.ts") && routePath === "/") {
|
|
return prefix;
|
|
}
|
|
if (file === "companies.ts" || file === "health.ts") {
|
|
return `${prefix}${routePath}`;
|
|
}
|
|
if (file === "auth.ts") {
|
|
return `${prefix}${routePath === "/" ? "" : routePath}`;
|
|
}
|
|
return `${prefix}${routePath}`;
|
|
}
|
|
|
|
function loadActualRoutes() {
|
|
const routes = new Set<string>();
|
|
const unknownRouteFiles: string[] = [];
|
|
|
|
for (const file of fs.readdirSync(ROUTES_DIR).filter((entry) => entry.endsWith(".ts"))) {
|
|
if (explicitOpenApiCoverageExclusions.has(file)) continue;
|
|
const prefix = apiPrefixes[file];
|
|
const source = fs.readFileSync(path.join(ROUTES_DIR, file), "utf8");
|
|
if (!prefix) {
|
|
if (ROUTER_METHOD_PATTERN.test(source)) {
|
|
unknownRouteFiles.push(file);
|
|
}
|
|
continue;
|
|
}
|
|
|
|
for (const match of source.matchAll(ROUTE_LITERAL_PATTERN)) {
|
|
const method = match[1].toUpperCase();
|
|
const routePath = match[2];
|
|
routes.add(`${method} ${normalizeExpressPath(resolveMountedPath(file, prefix, routePath))}`);
|
|
}
|
|
|
|
if (file === "companies.ts" && source.includes("router.post(COMPANY_IMPORT_ROUTE_PATH")) {
|
|
routes.add("POST /api/companies/import");
|
|
}
|
|
}
|
|
|
|
return { routes, unknownRouteFiles: unknownRouteFiles.sort() };
|
|
}
|
|
|
|
function loadSpecRoutes() {
|
|
const spec = buildOpenApiSpec();
|
|
const routes = new Set<string>();
|
|
|
|
for (const [routePath, pathItem] of Object.entries<Record<string, Record<string, unknown>>>(spec.paths ?? {})) {
|
|
for (const method of Object.keys(pathItem)) {
|
|
if (HTTP_METHODS.has(method)) {
|
|
routes.add(`${method.toUpperCase()} ${routePath}`);
|
|
}
|
|
}
|
|
}
|
|
|
|
return { spec, routes };
|
|
}
|
|
|
|
describe("openapi routes", () => {
|
|
it("serves the generated OpenAPI document", async () => {
|
|
const res = await request(createApp()).get("/api/openapi.json");
|
|
|
|
expect(res.status).toBe(200);
|
|
expect(res.body.openapi).toBe("3.0.0");
|
|
expect(res.body.info.title).toBe("Paperclip API");
|
|
expect(res.body.paths["/api/openapi.json"].get.summary).toBe("Get the generated OpenAPI document");
|
|
expect(res.body.paths["/api/companies/{companyId}/agents"].get.summary).toBe("List agents in a company");
|
|
expect(res.body.paths["/api/agents/{id}/keys"].post.summary).toBe("Create an agent API key");
|
|
expect(res.body.components.securitySchemes).toMatchObject({
|
|
BoardSessionAuth: { type: "apiKey", in: "cookie" },
|
|
BoardApiKeyAuth: { type: "http", scheme: "bearer" },
|
|
AgentBearerAuth: { type: "http", scheme: "bearer" },
|
|
});
|
|
expect(res.body.paths["/api/health"].get.security).toEqual([]);
|
|
expect(res.body.paths["/mcp/gateways/{gatewayPublicId}"].post.security).toEqual([]);
|
|
expect(res.body.paths["/api/mcp/gateways/{gatewayPublicId}"]).toBeUndefined();
|
|
expect(res.body.paths["/api/companies"].post.responses["201"]).toBeDefined();
|
|
expect(res.body.paths["/api/companies"].post.requestBody.content["application/json"].schema).toMatchObject({
|
|
type: "object",
|
|
properties: {
|
|
name: { type: "string", minLength: 1 },
|
|
},
|
|
required: ["name"],
|
|
});
|
|
expect(JSON.stringify(res.body.paths["/api/companies"].post.responses)).not.toContain("candidates");
|
|
expect(res.body.paths["/api/companies/{companyId}/skills/scan-projects"].post.responses["200"].content[
|
|
"application/json"
|
|
].schema).toMatchObject({
|
|
type: "object",
|
|
properties: {
|
|
candidates: { type: "array" },
|
|
},
|
|
required: expect.arrayContaining(["candidates"]),
|
|
});
|
|
expect(res.body.paths["/api/agents/{id}/keys"].post.requestBody.content["application/json"].schema).toMatchObject({
|
|
type: "object",
|
|
properties: {
|
|
name: { type: "string" },
|
|
},
|
|
});
|
|
expect(res.body.paths["/api/companies/{companyId}/folders"].post.responses["201"]).toBeDefined();
|
|
expect(res.body.paths["/api/companies/{companyId}/folders/items/move"].post.summary).toBe(
|
|
"Move an item into or out of a folder",
|
|
);
|
|
const createQueue = res.body.paths["/api/companies/{companyId}/decision-queues"].post;
|
|
expect(createQueue.security).toContainEqual({ AgentBearerAuth: [] });
|
|
expect(createQueue.responses["200"]).toBeDefined();
|
|
expect(createQueue.responses["201"]).toBeDefined();
|
|
expect(createQueue.requestBody.content["application/json"].schema).toMatchObject({
|
|
type: "object",
|
|
properties: {
|
|
key: { type: "string", minLength: 1, maxLength: 80 },
|
|
title: { type: "string", minLength: 1, maxLength: 120 },
|
|
},
|
|
required: ["key", "title"],
|
|
});
|
|
const updateTriage = res.body.paths[
|
|
"/api/companies/{companyId}/decision-triage/{sourceKind}/{sourceId}"
|
|
].put;
|
|
expect(updateTriage.responses["422"]).toBeDefined();
|
|
expect(updateTriage.requestBody.content["application/json"].schema.properties).toMatchObject({
|
|
decideBy: { nullable: true },
|
|
snoozedUntil: { type: "string", format: "date-time", nullable: true },
|
|
});
|
|
expect(JSON.stringify(res.body.paths["/api/tool-gateway/tools"].get)).not.toContain("sessionToken");
|
|
expect(JSON.stringify(res.body.paths["/api/tool-gateway/tools/call"].post)).not.toContain("sessionToken");
|
|
});
|
|
|
|
it("covers the mounted server routes exactly", () => {
|
|
const { routes: actualRoutes, unknownRouteFiles } = loadActualRoutes();
|
|
const { routes: specRoutes } = loadSpecRoutes();
|
|
|
|
const missingInSpec = [...actualRoutes].filter((route) => !specRoutes.has(route)).sort();
|
|
const extraInSpec = [...specRoutes].filter((route) => !actualRoutes.has(route)).sort();
|
|
|
|
expect({ unknownRouteFiles, missingInSpec, extraInSpec }).toEqual({
|
|
unknownRouteFiles: [],
|
|
missingInSpec: [],
|
|
extraInSpec: [],
|
|
});
|
|
});
|
|
|
|
it("documents auth and reviewed response-code invariants", () => {
|
|
const { spec } = loadSpecRoutes();
|
|
|
|
expect(spec.paths["/api/openapi.json"].get.security).toEqual([]);
|
|
expect(spec.paths["/api/plugins/install"].post.security).toEqual([
|
|
{ BoardSessionAuth: [] },
|
|
{ BoardApiKeyAuth: [] },
|
|
]);
|
|
expect(spec.paths["/api/plugins/install"].post["x-paperclip-authorization"]).toEqual({
|
|
actor: "board",
|
|
instanceAdmin: true,
|
|
});
|
|
expect(spec.paths["/api/execution-workspaces/{id}/reconcile-branch"].post.security).toEqual([
|
|
{ BoardSessionAuth: [] },
|
|
{ BoardApiKeyAuth: [] },
|
|
]);
|
|
expect(spec.paths["/api/execution-workspaces/{id}/reconcile-branch"].post["x-paperclip-authorization"]).toEqual({
|
|
actor: "board",
|
|
});
|
|
expect(spec.paths["/api/companies/{companyId}/cost-events"].post.responses["201"]).toBeDefined();
|
|
expect(spec.paths["/api/companies/{companyId}/cost-events"].post.responses["403"]).toBeDefined();
|
|
expect(spec.paths["/api/instance/database-backups"].post.responses["201"]).toBeDefined();
|
|
expect(spec.paths["/api/invites/{token}/accept"].post.responses["202"]).toBeDefined();
|
|
expect(spec.paths["/api/board-api-keys"].post.responses["201"]).toBeDefined();
|
|
expect(spec.paths["/api/companies/import"].post.responses["202"]).toBeDefined();
|
|
});
|
|
});
|