feat(search): add bulk extract endpoint (#9507)

## Thinking Path

> - Paperclip is the open source control plane people use to coordinate
AI-agent companies
> - Agents and operators need company-scoped search to discover relevant
issue history safely
> - The interactive search endpoint intentionally returns compact
excerpts and low pagination caps for UI use
> - Automation that inventories repeated references, such as
pull-request URLs, needs exhaustive distinct matches without loading
full issue objects into an LLM context
> - Client-provided regular expressions would create an unsafe and
expensive query surface, so extraction must remain literal with
server-owned expansion modes
> - This pull request adds a bounded agent-oriented extraction endpoint
with explicit truncation
> - The benefit is deterministic, compact bulk discovery across issues,
comments, and documents while preserving company authorization and rate
limits

## Linked Issues or Issue Description

### Subsystem affected

`server/` REST API and `packages/shared/` contracts.

### Problem or motivation

The existing interactive company search caps issue pagination and
snippets, so automation cannot reliably enumerate every distinct literal
or pull-request URL across issue descriptions, comments, and linked
documents without fetching large full issue payloads.

### Proposed solution

Add `GET /api/companies/:companyId/search/extract` with escaped literal
matching, optional server-owned URL token expansion,
issue/comment/document scopes, status/date filters, higher issue-level
pagination caps, compact source references, and explicit
pagination/match truncation flags.

### Alternatives considered

Reusing `GET /issues?q=` would return unnecessarily large issue objects;
increasing interactive-search snippet limits would make the UI API
heavier; accepting arbitrary client regex would expose avoidable
database cost and ReDoS risk.

### Roadmap alignment

`ROADMAP.md` does not currently list a conflicting company-search or
bulk-extraction initiative. GitHub searches found no directly
duplicative open issue or pull request.

## What Changed

- Added shared query validation and response contracts for literal and
URL extraction.
- Added a company-scoped extraction service that pages issues, gathers
matching issue/comment/document sources, expands URL tokens,
deduplicates values, and reports truncation explicitly.
- Added the authenticated route using the existing company-search
authorization decision and rate limiter.
- Added targeted Vitest coverage for URL extraction, multi-source
dedupe, date/status filters, match caps, cross-company denial, and rate
limiting.
- Documented the extraction surface in the implementation specification.

## Verification

- `pnpm exec vitest run
server/src/__tests__/company-search-extract-service.test.ts
server/src/__tests__/company-search-extract-routes.test.ts
server/src/__tests__/company-search-rate-limit-routes.test.ts
server/src/__tests__/company-search-service.test.ts` — 30 tests passed.
- `pnpm --filter @paperclipai/shared typecheck` — passed.
- `pnpm --filter @paperclipai/server typecheck` — passed.
- `git diff --check` — passed.

## Risks

- Bulk substring search can scan large text columns. The endpoint
mitigates this with a minimum literal length, bounded issue pagination,
a 20-distinct-match cap per issue, explicit truncation, existing
company-search rate limiting, and no client-provided regex.
- URL expansion uses a fixed server-owned pattern plus an escaped
literal. A security review is requested as part of PR review to confirm
the pattern and abuse controls.
- No database migration or existing API response shape changes are
included.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex CLI coding agent; exact runtime model ID and
context-window size were not exposed to the session. Tool-enabled code
execution and repository editing were used with medium reasoning effort.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/__tests__/company-search-extract-routes.test.ts

@@ -0,0 +1,96 @@
+import express from "express";
+import request from "supertest";
+import { describe, expect, it, vi } from "vitest";
+import type {
+  CompanySearchExtractQuery,
+  CompanySearchExtractResponse,
+  CompanySearchQuery,
+  CompanySearchResponse,
+} from "@paperclipai/shared";
+import { issueRoutes } from "../routes/issues.js";
+import { createCompanySearchRateLimiter } from "../services/company-search-rate-limit.js";
+
+function extractResponse(query: CompanySearchExtractQuery): CompanySearchExtractResponse {
+  return {
+    contains: query.contains,
+    kind: query.kind,
+    scope: query.scope,
+    limit: query.limit,
+    offset: query.offset,
+    results: [],
+    hasMore: false,
+    truncated: false,
+  };
+}
+
+function unusedSearch(_companyId: string, _query: CompanySearchQuery): Promise<CompanySearchResponse> {
+  throw new Error("interactive search should not be called");
+}
+
+function createApp(companyIds: string[], extract: (companyId: string, query: CompanySearchExtractQuery) => Promise<CompanySearchExtractResponse>) {
+  const app = express();
+  app.use((req, _res, next) => {
+    req.actor = {
+      type: "board",
+      userId: "user-1",
+      companyIds,
+      source: "session",
+      isInstanceAdmin: true,
+    };
+    next();
+  });
+  app.use("/api", issueRoutes({} as never, {} as never, {
+    searchService: { search: unusedSearch, extract },
+    searchRateLimiter: createCompanySearchRateLimiter({
+      maxRequests: 1,
+      windowMs: 60_000,
+      now: () => 1_000,
+    }),
+  }));
+  return app;
+}
+
+describe("company extract-search route", () => {
+  it("parses the extraction query and invokes the service", async () => {
+    const extract = vi.fn(async (_companyId: string, query: CompanySearchExtractQuery) => extractResponse(query));
+    const app = createApp(["company-1"], extract);
+
+    const response = await request(app)
+      .get("/api/companies/company-1/search/extract")
+      .query({ contains: "github.com/example/repo/pull", kind: "url", scope: "comments", limit: "200" })
+      .expect(200);
+
+    expect(extract).toHaveBeenCalledWith("company-1", expect.objectContaining({
+      contains: "github.com/example/repo/pull",
+      kind: "url",
+      scope: "comments",
+      limit: 200,
+    }));
+    expect(response.body).toMatchObject({ kind: "url", scope: "comments", truncated: false });
+  });
+
+  it("denies cross-company access before invoking the service", async () => {
+    const extract = vi.fn(async (_companyId: string, query: CompanySearchExtractQuery) => extractResponse(query));
+    const app = createApp(["company-1"], extract);
+
+    await request(app)
+      .get("/api/companies/company-2/search/extract?contains=needle")
+      .expect(403);
+
+    expect(extract).not.toHaveBeenCalled();
+  });
+
+  it("shares the company-search rate limiter", async () => {
+    const extract = vi.fn(async (_companyId: string, query: CompanySearchExtractQuery) => extractResponse(query));
+    const app = createApp(["company-1"], extract);
+
+    await request(app).get("/api/companies/company-1/search/extract?contains=needle").expect(200);
+    const limited = await request(app)
+      .get("/api/companies/company-1/search/extract?contains=needle")
+      .expect(429);
+
+    expect(extract).toHaveBeenCalledTimes(1);
+    expect(limited.body).toEqual({ error: "Search rate limit exceeded", retryAfterSeconds: 60 });
+    expect(limited.headers["retry-after"]).toBe("60");
+  });
+});

server/src/__tests__/company-search-extract-service.test.ts

@@ -0,0 +1,244 @@
+import { randomUUID } from "node:crypto";
+import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
+import {
+  companies,
+  createDb,
+  documents,
+  issueComments,
+  issueDocuments,
+  issues,
+} from "@paperclipai/db";
+import {
+  COMPANY_SEARCH_EXTRACT_MAX_MATCHES_PER_ISSUE,
+  companySearchExtractQuerySchema,
+} from "@paperclipai/shared";
+import {
+  getEmbeddedPostgresTestSupport,
+  startEmbeddedPostgresTestDatabase,
+} from "./helpers/embedded-postgres.js";
+import { companySearchExtractService } from "../services/company-search-extract.js";
+
+const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
+const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
+
+if (!embeddedPostgresSupport.supported) {
+  console.warn(
+    `Skipping embedded Postgres extract-search tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
+  );
+}
+
+describe("extract-search query validation", () => {
+  it("accepts supported extraction filters and rejects unsafe or ambiguous input", () => {
+    const parsed = companySearchExtractQuerySchema.parse({
+      contains: "github.com/paperclipai/paperclip/pull",
+      kind: "url",
+      scope: "comments",
+      status: "in_progress,in_review",
+      limit: "200",
+      offset: "5000",
+      updatedWithin: "30d",
+    });
+
+    expect(parsed.kind).toBe("url");
+    expect(parsed.scope).toBe("comments");
+    expect(parsed.status).toEqual(["in_progress", "in_review"]);
+    expect(() => companySearchExtractQuerySchema.parse({ contains: ".*", kind: "regex" })).toThrow();
+    expect(() => companySearchExtractQuerySchema.parse({ contains: "x" })).toThrow();
+    expect(() => companySearchExtractQuerySchema.parse({ contains: "needle", limit: "201" })).toThrow();
+    expect(() => companySearchExtractQuerySchema.parse({
+      contains: "needle",
+      updatedWithin: "30d",
+      updatedAfter: "2026-01-01T00:00:00.000Z",
+    })).toThrow();
+  });
+});
+
+describeEmbeddedPostgres("companySearchExtractService", () => {
+  let db!: ReturnType<typeof createDb>;
+  let svc!: ReturnType<typeof companySearchExtractService>;
+  let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
+
+  beforeAll(async () => {
+    tempDb = await startEmbeddedPostgresTestDatabase("paperclip-company-search-extract-");
+    db = createDb(tempDb.connectionString);
+    svc = companySearchExtractService(db);
+  }, 20_000);
+
+  afterEach(async () => {
+    await db.delete(issueDocuments);
+    await db.delete(documents);
+    await db.delete(issueComments);
+    await db.delete(issues);
+    await db.delete(companies);
+  });
+
+  afterAll(async () => {
+    await tempDb?.cleanup();
+  });
+
+  async function createCompany(name = "Paperclip") {
+    const companyId = randomUUID();
+    await db.insert(companies).values({
+      id: companyId,
+      name,
+      issuePrefix: `E${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
+      requireBoardApprovalForNewAgents: false,
+    });
+    return companyId;
+  }
+
+  async function createIssue(companyId: string, values: Partial<typeof issues.$inferInsert> = {}) {
+    const id = values.id ?? randomUUID();
+    await db.insert(issues).values({
+      id,
+      companyId,
+      identifier: values.identifier ?? "EXT-1",
+      title: values.title ?? "Extract target",
+      description: values.description ?? null,
+      status: values.status ?? "in_progress",
+      priority: values.priority ?? "medium",
+      ...values,
+    });
+    return id;
+  }
+
+  it("expands and deduplicates URLs across issue, comment, and document sources", async () => {
+    const companyId = await createCompany();
+    const firstUrl = "https://github.com/paperclipai/paperclip/pull/123";
+    const secondUrl = "https://github.com/paperclipai/paperclip/pull/456";
+    const thirdUrl = "https://github.com/paperclipai/paperclip/pull/789";
+    const issueId = await createIssue(companyId, {
+      description: `Primary ${firstUrl} and duplicate ${firstUrl}.`,
+    });
+    await db.insert(issueComments).values({
+      companyId,
+      issueId,
+      body: `Review ${secondUrl} and repeat ${firstUrl}`,
+    });
+    const documentId = randomUUID();
+    await db.insert(documents).values({
+      id: documentId,
+      companyId,
+      title: `PR notes ${thirdUrl}`,
+      latestBody: `Also see [the second PR](${secondUrl}).`,
+    });
+    await db.insert(issueDocuments).values({
+      companyId,
+      issueId,
+      documentId,
+      key: "plan",
+    });
+
+    const result = await svc.extract(companyId, companySearchExtractQuerySchema.parse({
+      contains: "github.com/paperclipai/paperclip/pull",
+      kind: "url",
+    }));
+
+    expect(result.results).toHaveLength(1);
+    expect(result.results[0]?.matches.map((match) => match.value)).toEqual([firstUrl, secondUrl, thirdUrl]);
+    expect(result.results[0]?.matches.map((match) => match.field)).toEqual([
+      "description",
+      "comment",
+      "document_title",
+    ]);
+    expect(result.results[0]?.matchesTruncated).toBe(false);
+    expect(result.truncated).toBe(false);
+  });
+
+  it("keeps URL sources selected by scheme-less queries", async () => {
+    const companyId = await createCompany();
+    const titleUrl = "https://github.com/paperclipai/paperclip/pull/101";
+    const descriptionUrl = "https://github.com/paperclipai/paperclip/pull/102";
+    const documentTitleUrl = "https://github.com/paperclipai/paperclip/pull/103";
+    const documentBodyUrl = "https://github.com/paperclipai/paperclip/pull/104";
+    const issueId = await createIssue(companyId, {
+      title: `Review ${titleUrl}`,
+      description: `Then merge ${descriptionUrl}`,
+    });
+    const documentId = randomUUID();
+    await db.insert(documents).values({
+      id: documentId,
+      companyId,
+      title: `Tracking ${documentTitleUrl}`,
+      latestBody: `Final follow-up ${documentBodyUrl}`,
+    });
+    await db.insert(issueDocuments).values({
+      companyId,
+      issueId,
+      documentId,
+      key: "plan",
+    });
+
+    const result = await svc.extract(companyId, companySearchExtractQuerySchema.parse({
+      contains: "github.com/paperclipai/paperclip/pull",
+      kind: "url",
+      scope: "all",
+    }));
+
+    expect(result.results[0]?.matches.map((match) => [match.field, match.value])).toEqual([
+      ["title", titleUrl],
+      ["description", descriptionUrl],
+      ["document_title", documentTitleUrl],
+      ["document_body", documentBodyUrl],
+    ]);
+  });
+
+  it("filters by issue update window and status", async () => {
+    const companyId = await createCompany();
+    const now = Date.now();
+    const recentId = await createIssue(companyId, {
+      identifier: "EXT-RECENT",
+      description: "needle",
+      status: "in_review",
+      updatedAt: new Date(now - 24 * 60 * 60 * 1000),
+    });
+    await createIssue(companyId, {
+      identifier: "EXT-OLD",
+      description: "needle",
+      status: "in_review",
+      updatedAt: new Date(now - 60 * 24 * 60 * 60 * 1000),
+    });
+    await createIssue(companyId, {
+      identifier: "EXT-DONE",
+      description: "needle",
+      status: "done",
+      updatedAt: new Date(now - 24 * 60 * 60 * 1000),
+    });
+
+    const result = await svc.extract(companyId, companySearchExtractQuerySchema.parse({
+      contains: "needle",
+      updatedWithin: "30d",
+      status: "in_review",
+    }));
+
+    expect(result.results.map((row) => row.issueId)).toEqual([recentId]);
+  });
+
+  it("caps distinct matches per issue and marks truncation explicitly", async () => {
+    const companyId = await createCompany();
+    const urls = Array.from(
+      { length: COMPANY_SEARCH_EXTRACT_MAX_MATCHES_PER_ISSUE + 1 },
+      (_, index) => `https://github.com/paperclipai/paperclip/pull/${index + 1}`,
+    );
+    await createIssue(companyId, { description: urls.join(" ") });
+
+    const result = await svc.extract(companyId, companySearchExtractQuerySchema.parse({
+      contains: "github.com/paperclipai/paperclip/pull",
+      kind: "url",
+    }));
+
+    expect(result.results[0]?.matches).toHaveLength(COMPANY_SEARCH_EXTRACT_MAX_MATCHES_PER_ISSUE);
+    expect(result.results[0]?.matchesTruncated).toBe(true);
+    expect(result.truncated).toBe(true);
+  });
+
+  it("does not return matching issues from another company", async () => {
+    const companyId = await createCompany();
+    const otherCompanyId = await createCompany("Other");
+    await createIssue(otherCompanyId, { description: "needle" });
+
+    const result = await svc.extract(companyId, companySearchExtractQuerySchema.parse({ contains: "needle" }));
+
+    expect(result.results).toEqual([]);
+  });
+});