mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 16:35:48 +01:00
External object references across issue surfaces (#8512)
## Thinking Path > - Paperclip is the open source control plane people use to coordinate AI agents, issues, approvals, comments, and work products. > - The involved subsystem is issue context: markdown links, issue properties, related work, lists, filters, inbox/sidebar status, and plugin-provided external context. > - The gap is that URLs to external systems currently remain mostly plain links, so humans and agents must manually open them to understand status, identity, and liveness. > - This matters because external work objects such as GitHub issues and pull requests are part of the operational state of a Paperclip company. > - The implementation keeps core provider-neutral: shared contracts, storage, sync, routes, and UI surfaces live in core while providers can contribute detection and status resolution. > - This pull request adds the external object reference foundation, GitHub provider support, issue-surface rendering, filters, sidebar/list/inbox signals, and test/story coverage. > - The benefit is that linked external work becomes inspectable Paperclip context without hardcoding every provider directly into the UI. ## Linked Issues or Issue Description No public GitHub issue exists for this work. Feature request: - Problem: URLs in Paperclip issues, comments, documents, and related surfaces do not expose provider status or object identity inline. - Proposed behavior: detect supported external object URLs, persist normalized references, refresh provider status, and render concise status-aware links across issue surfaces. - Users affected: board users, agents, and maintainers who triage issues containing external work links. - Acceptance: external object references are company-scoped, provider-extensible, visible in key issue surfaces, filterable where relevant, and covered by focused shared/server/UI tests. Related PR search: - No open duplicate PRs found for `external object references`. - Closed related prior attempt: #4556. ## What Changed - Added shared external-object contracts, validators, status/liveness helpers, and plugin protocol declarations. - Added database schema and additive migrations for external objects, source mentions, and display metadata. - Added server services/routes for detecting, syncing, summarizing, refreshing, and resolving external objects across issues, documents, comments, projects, and plugins. - Added a GitHub external-object provider plus plugin SDK authoring docs. - Wired UI presentation across markdown links, comments, issue chat, documents, properties, related work, issue rows, filters, inbox/sidebar badges, and Storybook stories. - Rebasing cleanup: moved the branch onto current `master`, repaired stale worktree provision config, hardened environment-sensitive tests/mocks, and removed committed screenshot artifacts from the PR branch to keep the reviewable file set below tool limits. ## Verification - `pnpm exec vitest run packages/shared/src/external-objects.test.ts server/src/__tests__/external-object-routes.test.ts server/src/__tests__/external-objects-service.test.ts ui/src/components/ExternalObjectPill.test.tsx ui/src/lib/external-objects.test.ts` passed after rebasing: 5 files, 56 tests. - Historical branch verification before this PR creation included `pnpm test:run`, `pnpm -r typecheck`, and `pnpm build`; this PR body does not claim those were rerun after the final rebase. ## Risks - Medium: this adds a new cross-surface sync path on issue/document/comment writes. The implementation uses safe sync wrappers so external-object failures warn instead of blocking core mutations. - Medium: the migrations introduce new tables and indexes. They are additive and company-scoped. - Medium: provider-specific URL parsing can miss or misclassify edge cases. Shared canonicalization tests and provider tests cover current GitHub shapes. - Low: UI badge/filter behavior could add visual noise for object-heavy issues; component tests and Storybook stories cover the intended surfaces. > Roadmap checked: `ROADMAP.md` references the plugin system as the current extension path and does not list a duplicate core feature. Related long-range docs discuss external references, work products, preview URLs, and plugin extension points; this PR implements the scoped external-object reference foundation. ## Model Used OpenAI Codex, GPT-5 coding-agent runtime, with shell and GitHub CLI tool use. Reasoning mode: medium. Exact deployed runtime model ID and context window were not exposed in the environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
cd38c150b0
commit
2dbaf4a7fa
97 files changed
+8217
-84
No files matched your search
@@ -180,5 +180,5 @@ describeLiveSsh("live SSH environment smoke", () => {
|
||||
expect(result.stdout).toContain(config.remoteWorkspacePath);
|
||||
expect(result.stdout).toContain("git");
|
||||
expect(result.stdout).toContain("tar");
|
||||
});
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -0,0 +1,252 @@
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const issueId = "11111111-1111-4111-8111-111111111111";
|
||||
const companyId = "22222222-2222-4222-8222-222222222222";
|
||||
const ownerAgentId = "33333333-3333-4333-8333-333333333333";
|
||||
const peerAgentId = "44444444-4444-4444-8444-444444444444";
|
||||
const ownerRunId = "55555555-5555-4555-8555-555555555555";
|
||||
|
||||
const mockIssueService = vi.hoisted(() => ({
|
||||
assertCheckoutOwner: vi.fn(),
|
||||
getById: vi.fn(),
|
||||
}));
|
||||
|
||||
const mockAccessService = vi.hoisted(() => ({
|
||||
decide: vi.fn(),
|
||||
hasPermission: vi.fn(),
|
||||
}));
|
||||
|
||||
const mockAgentService = vi.hoisted(() => ({
|
||||
list: vi.fn(),
|
||||
}));
|
||||
|
||||
const mockExternalObjectsService = vi.hoisted(() => ({
|
||||
getIssueSummary: vi.fn(),
|
||||
getIssueSummaries: vi.fn(),
|
||||
listForIssue: vi.fn(),
|
||||
refreshIssueObjects: vi.fn(),
|
||||
}));
|
||||
const mockInstanceSettingsService = vi.hoisted(() => ({
|
||||
getExperimental: vi.fn(),
|
||||
}));
|
||||
|
||||
function registerRouteMocks() {
|
||||
vi.doMock("../services/external-objects.js", () => ({
|
||||
externalObjectService: () => mockExternalObjectsService,
|
||||
}));
|
||||
|
||||
vi.doMock("../services/instance-settings.js", () => ({
|
||||
instanceSettingsService: () => mockInstanceSettingsService,
|
||||
}));
|
||||
|
||||
vi.doMock("../services/task-watchdog-scope.js", () => ({
|
||||
TASK_WATCHDOG_ORIGIN_KIND: "task_watchdog",
|
||||
resolveTaskWatchdogMutationScope: vi.fn(async () => ({ kind: "none" })),
|
||||
taskWatchdogScopeAllowsIssueMutation: vi.fn(async () => ({ kind: "none" })),
|
||||
}));
|
||||
|
||||
vi.doMock("../services/index.js", () => ({
|
||||
accessService: () => mockAccessService,
|
||||
agentService: () => mockAgentService,
|
||||
companyService: () => ({
|
||||
getById: vi.fn(async () => null),
|
||||
}),
|
||||
companySearchService: () => ({}),
|
||||
documentAnnotationService: () => ({}),
|
||||
documentService: () => ({}),
|
||||
executionWorkspaceService: () => ({}),
|
||||
feedbackService: () => ({}),
|
||||
goalService: () => ({}),
|
||||
heartbeatService: () => ({
|
||||
wakeup: vi.fn(async () => undefined),
|
||||
reportRunActivity: vi.fn(async () => undefined),
|
||||
getRun: vi.fn(async () => null),
|
||||
getActiveRunForAgent: vi.fn(async () => null),
|
||||
cancelRun: vi.fn(async () => null),
|
||||
}),
|
||||
issueApprovalService: () => ({}),
|
||||
issueRecoveryActionService: () => ({}),
|
||||
issueReferenceService: () => ({
|
||||
listIssueReferenceSummary: async () => ({ outbound: [], inbound: [] }),
|
||||
}),
|
||||
issueService: () => mockIssueService,
|
||||
issueThreadInteractionService: () => ({}),
|
||||
logActivity: vi.fn(async () => undefined),
|
||||
projectService: () => ({}),
|
||||
routineService: () => ({}),
|
||||
workProductService: () => ({}),
|
||||
}));
|
||||
}
|
||||
|
||||
function makeIssue(overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
id: issueId,
|
||||
companyId,
|
||||
status: "in_progress",
|
||||
priority: "medium",
|
||||
projectId: null,
|
||||
goalId: null,
|
||||
parentId: null,
|
||||
assigneeAgentId: ownerAgentId,
|
||||
assigneeUserId: null,
|
||||
identifier: "PAP-2265",
|
||||
title: "External object routes",
|
||||
executionWorkspaceId: null,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
async function createApp(actor: Express.Request["actor"]) {
|
||||
const [{ errorHandler }, { issueRoutes }] = await Promise.all([
|
||||
vi.importActual<typeof import("../middleware/index.js")>("../middleware/index.js"),
|
||||
vi.importActual<typeof import("../routes/issues.js")>("../routes/issues.js"),
|
||||
]);
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
req.actor = actor;
|
||||
next();
|
||||
});
|
||||
app.use("/api", issueRoutes({} as any, { provider: "local_disk" } as any));
|
||||
app.use(errorHandler);
|
||||
return app;
|
||||
}
|
||||
|
||||
function boardActor(): Express.Request["actor"] {
|
||||
return {
|
||||
type: "board",
|
||||
userId: "board-user",
|
||||
userName: null,
|
||||
userEmail: null,
|
||||
companyIds: [companyId],
|
||||
memberships: [],
|
||||
isInstanceAdmin: false,
|
||||
source: "local_implicit",
|
||||
};
|
||||
}
|
||||
|
||||
function ownerActor(): Express.Request["actor"] {
|
||||
return {
|
||||
type: "agent",
|
||||
agentId: ownerAgentId,
|
||||
companyId,
|
||||
keyId: "key-1",
|
||||
runId: ownerRunId,
|
||||
source: "agent_key",
|
||||
};
|
||||
}
|
||||
|
||||
function peerActor(): Express.Request["actor"] {
|
||||
return {
|
||||
type: "agent",
|
||||
agentId: peerAgentId,
|
||||
companyId,
|
||||
keyId: "key-2",
|
||||
runId: "66666666-6666-4666-8666-666666666666",
|
||||
source: "agent_key",
|
||||
};
|
||||
}
|
||||
|
||||
describe("external object routes", () => {
|
||||
beforeEach(() => {
|
||||
vi.resetModules();
|
||||
vi.doUnmock("../routes/issues.js");
|
||||
vi.doUnmock("../services/index.js");
|
||||
vi.doUnmock("../services/external-objects.js");
|
||||
registerRouteMocks();
|
||||
vi.resetAllMocks();
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue());
|
||||
mockIssueService.assertCheckoutOwner.mockResolvedValue({ adoptedFromRunId: null });
|
||||
mockAccessService.hasPermission.mockResolvedValue(false);
|
||||
mockAccessService.decide.mockImplementation(async ({ action }: { action: string }) => ({
|
||||
allowed: action === "issue:mutate",
|
||||
explanation: "Denied by test mock",
|
||||
}));
|
||||
mockAgentService.list.mockResolvedValue([
|
||||
{ id: ownerAgentId, companyId, reportsTo: null, permissions: { canCreateAgents: false } },
|
||||
{ id: peerAgentId, companyId, reportsTo: null, permissions: { canCreateAgents: false } },
|
||||
]);
|
||||
mockExternalObjectsService.getIssueSummary.mockResolvedValue({ total: 1, objects: [] });
|
||||
mockExternalObjectsService.getIssueSummaries.mockResolvedValue(new Map([
|
||||
[issueId, { total: 1, objects: [] }],
|
||||
]));
|
||||
mockExternalObjectsService.listForIssue.mockResolvedValue([]);
|
||||
mockExternalObjectsService.refreshIssueObjects.mockResolvedValue([
|
||||
{ object: { id: "77777777-7777-4777-8777-777777777777" }, refreshed: false, reason: "no_resolver" },
|
||||
]);
|
||||
mockInstanceSettingsService.getExperimental.mockResolvedValue({
|
||||
enableExternalObjects: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("enforces company access on read routes", async () => {
|
||||
const app = await createApp({ ...ownerActor(), companyId: "other-company" });
|
||||
|
||||
const res = await request(app).get(`/api/issues/${issueId}/external-object-summary`);
|
||||
|
||||
expect(res.status).toBe(403);
|
||||
expect(mockExternalObjectsService.getIssueSummary).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("allows board users to read issue external object summaries", async () => {
|
||||
const app = await createApp(boardActor());
|
||||
|
||||
const res = await request(app).get(`/api/issues/${issueId}/external-object-summary`);
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.total).toBe(1);
|
||||
expect(mockExternalObjectsService.getIssueSummary).toHaveBeenCalledWith(issueId);
|
||||
});
|
||||
|
||||
it("allows board users to fetch company-scoped external object summaries in bulk", async () => {
|
||||
const app = await createApp(boardActor());
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/companies/${companyId}/issues/external-object-summaries`)
|
||||
.send({ issueIds: [issueId] });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.summaries[issueId].total).toBe(1);
|
||||
expect(mockExternalObjectsService.getIssueSummaries).toHaveBeenCalledWith(companyId, [issueId]);
|
||||
});
|
||||
|
||||
it("enforces company access on bulk external object summaries", async () => {
|
||||
const app = await createApp({ ...ownerActor(), companyId: "other-company" });
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/companies/${companyId}/issues/external-object-summaries`)
|
||||
.send({ issueIds: [issueId] });
|
||||
|
||||
expect(res.status).toBe(403);
|
||||
expect(mockExternalObjectsService.getIssueSummaries).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("requires active checkout ownership for agent manual refresh", async () => {
|
||||
const app = await createApp(peerActor());
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/issues/${issueId}/external-objects/refresh`)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBe(409);
|
||||
expect(res.body.error).toBe("Issue is checked out by another agent");
|
||||
expect(mockExternalObjectsService.refreshIssueObjects).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("allows the checked-out agent to request manual refresh", async () => {
|
||||
const app = await createApp(ownerActor());
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/issues/${issueId}/external-objects/refresh`)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(mockIssueService.assertCheckoutOwner).toHaveBeenCalledWith(issueId, ownerAgentId, ownerRunId);
|
||||
expect(mockExternalObjectsService.refreshIssueObjects).toHaveBeenCalledWith(issueId, expect.objectContaining({
|
||||
companyId,
|
||||
actor: expect.objectContaining({ actorType: "agent", actorId: ownerAgentId }),
|
||||
}));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,705 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import {
|
||||
companies,
|
||||
createDb,
|
||||
activityLog,
|
||||
externalObjectMentions,
|
||||
externalObjects,
|
||||
issueComments,
|
||||
issues,
|
||||
plugins,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import {
|
||||
createExternalObjectDetectorRegistry,
|
||||
createExternalObjectResolverRegistry,
|
||||
externalObjectService,
|
||||
type ExternalObjectResolver,
|
||||
} from "../services/external-objects.js";
|
||||
import { canonicalizeExternalObjectUrl } from "@paperclipai/shared/external-objects-server";
|
||||
import type { PaperclipPluginManifestV1 } from "@paperclipai/shared";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { createGitHubExternalObjectProvider } from "../services/github-external-object-provider.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping embedded Postgres external object tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
describe("external object registries", () => {
|
||||
it("lets provider detectors claim urls before the generic fallback", async () => {
|
||||
const canonical = canonicalizeExternalObjectUrl("https://github.com/acme/app/pull/42");
|
||||
if (!canonical) throw new Error("expected canonical url");
|
||||
const registry = createExternalObjectDetectorRegistry([
|
||||
{
|
||||
key: "github",
|
||||
detect: ({ urls }) =>
|
||||
urls.map((url) => ({
|
||||
canonical: url,
|
||||
detectorKey: "github",
|
||||
providerKey: "github",
|
||||
objectType: "pull_request",
|
||||
externalId: "acme/app#42",
|
||||
confidence: "exact",
|
||||
})),
|
||||
},
|
||||
]);
|
||||
|
||||
const detections = await registry.detect({
|
||||
companyId: "company-1",
|
||||
urls: [canonical],
|
||||
sourceContext: {
|
||||
companyId: "company-1",
|
||||
sourceIssueId: "issue-1",
|
||||
sourceKind: "description",
|
||||
sourceRecordId: null,
|
||||
documentKey: null,
|
||||
propertyKey: null,
|
||||
},
|
||||
});
|
||||
|
||||
expect(detections).toHaveLength(1);
|
||||
expect(detections[0]).toMatchObject({
|
||||
providerKey: "github",
|
||||
objectType: "pull_request",
|
||||
externalId: "acme/app#42",
|
||||
});
|
||||
});
|
||||
|
||||
it("falls back to generic url objects when no provider detector claims a url", async () => {
|
||||
const canonical = canonicalizeExternalObjectUrl("https://example.com/path?token=secret#frag");
|
||||
if (!canonical) throw new Error("expected canonical url");
|
||||
const registry = createExternalObjectDetectorRegistry([]);
|
||||
|
||||
const detections = await registry.detect({
|
||||
companyId: "company-1",
|
||||
urls: [canonical],
|
||||
sourceContext: {
|
||||
companyId: "company-1",
|
||||
sourceIssueId: "issue-1",
|
||||
sourceKind: "description",
|
||||
sourceRecordId: null,
|
||||
documentKey: null,
|
||||
propertyKey: null,
|
||||
},
|
||||
});
|
||||
|
||||
expect(detections[0]).toMatchObject({
|
||||
providerKey: "url",
|
||||
objectType: "link",
|
||||
externalId: canonical.canonicalIdentityHash,
|
||||
displayTitle: "https://example.com/path",
|
||||
});
|
||||
});
|
||||
|
||||
it("matches resolvers by provider and optional object type", () => {
|
||||
const fallbackResolver: ExternalObjectResolver = {
|
||||
providerKey: "github",
|
||||
resolve: async () => ({
|
||||
ok: true,
|
||||
snapshot: { statusCategory: "unknown", statusTone: "neutral" },
|
||||
}),
|
||||
};
|
||||
const pullRequestResolver: ExternalObjectResolver = {
|
||||
providerKey: "github",
|
||||
objectType: "pull_request",
|
||||
resolve: async () => ({
|
||||
ok: true,
|
||||
snapshot: { statusCategory: "open", statusTone: "info" },
|
||||
}),
|
||||
};
|
||||
const registry = createExternalObjectResolverRegistry([pullRequestResolver, fallbackResolver]);
|
||||
|
||||
expect(registry.find({ providerKey: "github", objectType: "pull_request" })).toBe(pullRequestResolver);
|
||||
expect(registry.find({ providerKey: "github", objectType: "issue" })).toBe(fallbackResolver);
|
||||
expect(registry.find({ providerKey: "linear", objectType: "issue" })).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("GitHub external object provider", () => {
|
||||
function githubObject(path: string, objectType: "pull_request" | "issue") {
|
||||
const canonical = canonicalizeExternalObjectUrl(`https://github.com/acme/app/${path}`);
|
||||
if (!canonical) throw new Error("expected canonical url");
|
||||
return {
|
||||
id: randomUUID(),
|
||||
companyId: "company-1",
|
||||
providerKey: "github",
|
||||
objectType,
|
||||
externalId: `acme/app#${path}`,
|
||||
sanitizedCanonicalUrl: canonical.sanitizedCanonicalUrl,
|
||||
canonicalIdentityHash: canonical.canonicalIdentityHash,
|
||||
displayTitle: "acme/app#42",
|
||||
statusKey: null,
|
||||
statusLabel: null,
|
||||
statusCategory: "unknown",
|
||||
statusTone: "neutral",
|
||||
liveness: "unknown",
|
||||
isTerminal: false,
|
||||
data: {},
|
||||
remoteVersion: null,
|
||||
etag: null,
|
||||
} as any;
|
||||
}
|
||||
|
||||
function response(body: Record<string, unknown>, init: ResponseInit = {}) {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json", etag: '"etag-1"', ...(init.headers ?? {}) },
|
||||
...init,
|
||||
});
|
||||
}
|
||||
|
||||
it("detects GitHub pull request and issue URLs before the generic fallback", async () => {
|
||||
const provider = createGitHubExternalObjectProvider({} as any, { tokenProvider: null });
|
||||
const pr = canonicalizeExternalObjectUrl("https://github.com/Acme/App/pull/42?token=secret#discussion");
|
||||
const issue = canonicalizeExternalObjectUrl("https://github.com/Acme/App/issues/7");
|
||||
const other = canonicalizeExternalObjectUrl("https://example.com/Acme/App/pull/42");
|
||||
if (!pr || !issue || !other) throw new Error("expected canonical urls");
|
||||
|
||||
const detections = await provider.detector.detect({
|
||||
companyId: "company-1",
|
||||
urls: [pr, issue, other],
|
||||
sourceContext: {
|
||||
companyId: "company-1",
|
||||
sourceIssueId: "issue-1",
|
||||
sourceKind: "description",
|
||||
sourceRecordId: null,
|
||||
documentKey: null,
|
||||
propertyKey: null,
|
||||
},
|
||||
});
|
||||
|
||||
expect(detections).toEqual([
|
||||
expect.objectContaining({
|
||||
providerKey: "github",
|
||||
objectType: "pull_request",
|
||||
externalId: "acme/app#pull/42",
|
||||
displayKey: "GitHub Pull Request",
|
||||
iconKey: "github",
|
||||
displayTitle: "Acme/App#42",
|
||||
}),
|
||||
expect.objectContaining({
|
||||
providerKey: "github",
|
||||
objectType: "issue",
|
||||
externalId: "acme/app#issues/7",
|
||||
displayKey: "GitHub Issue",
|
||||
iconKey: "github",
|
||||
displayTitle: "Acme/App#7",
|
||||
}),
|
||||
]);
|
||||
expect(JSON.stringify(detections)).not.toContain("secret");
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
"open",
|
||||
{ state: "open", draft: false, merged: false, title: "Ship it", updated_at: "2026-04-24T01:02:03Z" },
|
||||
{ statusKey: "open", statusLabel: "Open", statusIconKey: "git-pull-request", statusCategory: "open", statusTone: "info", isTerminal: false },
|
||||
],
|
||||
[
|
||||
"draft",
|
||||
{ state: "open", draft: true, merged: false, title: "WIP", updated_at: "2026-04-24T01:02:03Z" },
|
||||
{ statusKey: "draft", statusLabel: "Draft", statusIconKey: "clock", statusCategory: "waiting", statusTone: "warning", isTerminal: false },
|
||||
],
|
||||
[
|
||||
"closed",
|
||||
{ state: "closed", draft: false, merged: false, title: "Closed PR", updated_at: "2026-04-24T01:02:03Z" },
|
||||
{ statusKey: "closed", statusLabel: "Closed", statusIconKey: "x-circle", statusCategory: "closed", statusTone: "muted", isTerminal: true },
|
||||
],
|
||||
[
|
||||
"merged",
|
||||
{ state: "closed", draft: false, merged: true, title: "Merged PR", updated_at: "2026-04-24T01:02:03Z" },
|
||||
{ statusKey: "merged", statusLabel: "Merged", statusIconKey: "git-merge", statusCategory: "succeeded", statusTone: "success", isTerminal: true },
|
||||
],
|
||||
])("resolves a %s pull request snapshot", async (_name, body, expected) => {
|
||||
const fetch = vi.fn(async () => response(body));
|
||||
const provider = createGitHubExternalObjectProvider({} as any, { fetch, tokenProvider: null });
|
||||
const resolver = provider.resolvers.find((entry) => entry.objectType === "pull_request")!;
|
||||
|
||||
const result = await resolver.resolve({
|
||||
companyId: "company-1",
|
||||
object: githubObject("pull/42", "pull_request"),
|
||||
});
|
||||
|
||||
expect(fetch).toHaveBeenCalledWith(
|
||||
"https://api.github.com/repos/acme/app/pulls/42",
|
||||
expect.objectContaining({
|
||||
headers: expect.not.objectContaining({ authorization: expect.any(String) }),
|
||||
}),
|
||||
);
|
||||
expect(result).toEqual({
|
||||
ok: true,
|
||||
snapshot: expect.objectContaining({
|
||||
...expected,
|
||||
displayKey: "GitHub Pull Request",
|
||||
iconKey: "github",
|
||||
displayTitle: expect.stringContaining(String(body.title)),
|
||||
remoteVersion: "2026-04-24T01:02:03Z",
|
||||
etag: '"etag-1"',
|
||||
data: expect.objectContaining({
|
||||
provider: "github",
|
||||
owner: "acme",
|
||||
repo: "app",
|
||||
number: 42,
|
||||
}),
|
||||
}),
|
||||
});
|
||||
expect(JSON.stringify(result)).not.toContain("authorization");
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
"open",
|
||||
{ state: "open", title: "Issue", state_reason: null, updated_at: "2026-04-24T01:02:03Z" },
|
||||
{ statusKey: "open", statusLabel: "Open", statusIconKey: "circle-dot", statusCategory: "open", statusTone: "info", isTerminal: false },
|
||||
],
|
||||
[
|
||||
"closed",
|
||||
{ state: "closed", title: "Issue", state_reason: "completed", updated_at: "2026-04-24T01:02:03Z" },
|
||||
{ statusKey: "closed_completed", statusLabel: "Closed: completed", statusIconKey: "circle", statusCategory: "closed", statusTone: "muted", isTerminal: true },
|
||||
],
|
||||
])("resolves a %s issue snapshot", async (_name, body, expected) => {
|
||||
const fetch = vi.fn(async () => response(body));
|
||||
const provider = createGitHubExternalObjectProvider({} as any, { fetch, tokenProvider: null });
|
||||
const resolver = provider.resolvers.find((entry) => entry.objectType === "issue")!;
|
||||
|
||||
const result = await resolver.resolve({
|
||||
companyId: "company-1",
|
||||
object: githubObject("issues/42", "issue"),
|
||||
});
|
||||
|
||||
expect(fetch).toHaveBeenCalledWith("https://api.github.com/repos/acme/app/issues/42", expect.any(Object));
|
||||
expect(result).toEqual({
|
||||
ok: true,
|
||||
snapshot: expect.objectContaining({
|
||||
...expected,
|
||||
displayKey: "GitHub Issue",
|
||||
iconKey: "github",
|
||||
data: expect.objectContaining({
|
||||
provider: "github",
|
||||
owner: "acme",
|
||||
repo: "app",
|
||||
number: 42,
|
||||
}),
|
||||
}),
|
||||
});
|
||||
});
|
||||
|
||||
it("uses a configured token without storing it in the resolved snapshot", async () => {
|
||||
const fetch = vi.fn(async (_url: string, init?: RequestInit) => {
|
||||
expect(init?.headers).toEqual(expect.objectContaining({ authorization: "Bearer ghp_secret" }));
|
||||
return response({ state: "open", draft: false, merged: false, title: "Private PR" });
|
||||
});
|
||||
const provider = createGitHubExternalObjectProvider({} as any, {
|
||||
fetch,
|
||||
tokenProvider: async () => "ghp_secret",
|
||||
});
|
||||
const resolver = provider.resolvers.find((entry) => entry.objectType === "pull_request")!;
|
||||
|
||||
const result = await resolver.resolve({
|
||||
companyId: "company-1",
|
||||
object: githubObject("pull/42", "pull_request"),
|
||||
});
|
||||
|
||||
expect(result.ok).toBe(true);
|
||||
expect(JSON.stringify(result)).not.toContain("ghp_secret");
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
"auth-required",
|
||||
new Response("", { status: 401 }),
|
||||
{ ok: false, liveness: "auth_required", errorCode: "github_auth_required" },
|
||||
],
|
||||
[
|
||||
"rate-limit",
|
||||
new Response("", {
|
||||
status: 403,
|
||||
headers: { "x-ratelimit-remaining": "0", "x-ratelimit-reset": String(Math.floor(Date.now() / 1000) + 120) },
|
||||
}),
|
||||
{ ok: false, liveness: "unreachable", errorCode: "github_rate_limited" },
|
||||
],
|
||||
[
|
||||
"not-found",
|
||||
new Response("", { status: 404, headers: { etag: '"missing"' } }),
|
||||
{ ok: true, snapshot: expect.objectContaining({ displayKey: "GitHub Pull Request", iconKey: "github", statusKey: "not_found", statusIconKey: "archive", statusCategory: "archived", statusTone: "muted" }) },
|
||||
],
|
||||
])("maps %s responses to provider-safe results", async (_name, githubResponse, expected) => {
|
||||
const provider = createGitHubExternalObjectProvider({} as any, {
|
||||
fetch: async () => githubResponse,
|
||||
tokenProvider: null,
|
||||
});
|
||||
const resolver = provider.resolvers.find((entry) => entry.objectType === "pull_request")!;
|
||||
|
||||
const result = await resolver.resolve({
|
||||
companyId: "company-1",
|
||||
object: githubObject("pull/42", "pull_request"),
|
||||
});
|
||||
|
||||
expect(result).toEqual(expect.objectContaining(expected));
|
||||
expect(JSON.stringify(result)).not.toContain("http");
|
||||
});
|
||||
});
|
||||
|
||||
describeEmbeddedPostgres("externalObjectService", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-external-objects-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await db.delete(activityLog);
|
||||
await db.delete(externalObjectMentions);
|
||||
await db.delete(externalObjects);
|
||||
await db.delete(issueComments);
|
||||
await db.delete(issues);
|
||||
await db.delete(plugins);
|
||||
await db.delete(companies);
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
async function createIssue(companyId = randomUUID()) {
|
||||
const issueId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `E${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(issues).values({
|
||||
id: issueId,
|
||||
companyId,
|
||||
identifier: `PAP-${companyId.replace(/-/g, "").slice(0, 12).toUpperCase()}`,
|
||||
title: "External refs",
|
||||
description: "Track https://github.com/acme/app/pull/42?token=secret#discussion twice https://github.com/acme/app/pull/42.",
|
||||
status: "todo",
|
||||
priority: "medium",
|
||||
});
|
||||
return { companyId, issueId };
|
||||
}
|
||||
|
||||
it("syncs sanitized, deduped mentions without storing secret-bearing urls", async () => {
|
||||
const { companyId, issueId } = await createIssue();
|
||||
const svc = externalObjectService(db);
|
||||
|
||||
await svc.syncIssue(issueId);
|
||||
|
||||
const [objectRows, mentionRows] = await Promise.all([
|
||||
db.select().from(externalObjects),
|
||||
db.select().from(externalObjectMentions),
|
||||
]);
|
||||
expect(objectRows).toHaveLength(1);
|
||||
expect(objectRows[0]).toMatchObject({
|
||||
companyId,
|
||||
providerKey: "github",
|
||||
objectType: "pull_request",
|
||||
externalId: "acme/app#pull/42",
|
||||
sanitizedCanonicalUrl: "https://github.com/acme/app/pull/42",
|
||||
liveness: "unknown",
|
||||
statusCategory: "unknown",
|
||||
});
|
||||
expect(JSON.stringify(objectRows[0])).not.toContain("secret");
|
||||
expect(mentionRows).toHaveLength(1);
|
||||
expect(mentionRows[0]).toMatchObject({
|
||||
companyId,
|
||||
sourceIssueId: issueId,
|
||||
sourceKind: "description",
|
||||
sanitizedDisplayUrl: "https://github.com/acme/app/pull/42",
|
||||
matchedTextRedacted: "https://github.com/acme/app/pull/42",
|
||||
});
|
||||
});
|
||||
|
||||
it("no-ops detection and summaries when external objects are disabled", async () => {
|
||||
const { issueId } = await createIssue();
|
||||
const svc = externalObjectService(db, { enabled: false });
|
||||
|
||||
await svc.syncIssue(issueId);
|
||||
|
||||
const [objectRows, mentionRows, summary] = await Promise.all([
|
||||
db.select().from(externalObjects),
|
||||
db.select().from(externalObjectMentions),
|
||||
svc.getIssueSummary(issueId),
|
||||
]);
|
||||
expect(objectRows).toHaveLength(0);
|
||||
expect(mentionRows).toHaveLength(0);
|
||||
expect(summary).toMatchObject({
|
||||
total: 0,
|
||||
byStatusCategory: {},
|
||||
byLiveness: {},
|
||||
highestSeverity: "neutral",
|
||||
objects: [],
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves last-known status when resolver reports auth and unreachable failures", async () => {
|
||||
const { companyId, issueId } = await createIssue();
|
||||
const resolver: ExternalObjectResolver = {
|
||||
providerKey: "url",
|
||||
objectType: "link",
|
||||
resolve: vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce({
|
||||
ok: true,
|
||||
snapshot: {
|
||||
statusCategory: "open",
|
||||
statusTone: "info",
|
||||
statusKey: "open",
|
||||
statusLabel: "Open",
|
||||
ttlSeconds: 1,
|
||||
},
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
ok: false,
|
||||
liveness: "auth_required",
|
||||
errorCode: "auth_required",
|
||||
errorMessage: "token=secret failed for https://github.com/acme/app/pull/42?token=secret",
|
||||
retryAfterSeconds: 60,
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
ok: false,
|
||||
liveness: "unreachable",
|
||||
errorCode: "network",
|
||||
errorMessage: "GET https://github.com/acme/app/pull/42 failed",
|
||||
retryAfterSeconds: 60,
|
||||
}),
|
||||
};
|
||||
const svc = externalObjectService(db, { resolvers: [resolver], github: false });
|
||||
await svc.syncIssue(issueId);
|
||||
const object = await db.select().from(externalObjects).then((rows) => rows[0]!);
|
||||
|
||||
await svc.refreshObject(object.id, { companyId, force: true });
|
||||
await svc.refreshObject(object.id, { companyId, force: true });
|
||||
|
||||
const authFailure = await db.select().from(externalObjects).then((rows) => rows[0]!);
|
||||
expect(authFailure.lastErrorMessage).toContain("token=[redacted]");
|
||||
expect(authFailure.lastErrorMessage).not.toContain("secret");
|
||||
|
||||
await svc.refreshObject(object.id, { companyId, force: true });
|
||||
|
||||
const updated = await db.select().from(externalObjects).then((rows) => rows[0]!);
|
||||
expect(updated.statusCategory).toBe("open");
|
||||
expect(updated.statusLabel).toBe("Open");
|
||||
expect(updated.liveness).toBe("unreachable");
|
||||
expect(updated.lastErrorMessage).toContain("[redacted-url]");
|
||||
expect(updated.lastErrorMessage).not.toContain("secret");
|
||||
});
|
||||
|
||||
it("schedules newly detected objects for automatic refresh", async () => {
|
||||
const { companyId, issueId } = await createIssue();
|
||||
const resolve = vi.fn(async () => ({
|
||||
ok: true as const,
|
||||
snapshot: {
|
||||
statusCategory: "open" as const,
|
||||
statusTone: "info" as const,
|
||||
statusKey: "open",
|
||||
statusLabel: "Open",
|
||||
ttlSeconds: 300,
|
||||
},
|
||||
}));
|
||||
const resolver: ExternalObjectResolver = {
|
||||
providerKey: "url",
|
||||
objectType: "link",
|
||||
resolve,
|
||||
};
|
||||
const svc = externalObjectService(db, { resolvers: [resolver], github: false });
|
||||
await svc.syncIssue(issueId);
|
||||
const object = await db.select().from(externalObjects).then((rows) => rows[0]!);
|
||||
|
||||
expect(object.nextRefreshAt).toBeInstanceOf(Date);
|
||||
|
||||
const refreshed = await svc.refreshDueObjects(companyId, 50, new Date(Date.now() + 1_000));
|
||||
|
||||
expect(refreshed).toHaveLength(1);
|
||||
expect(resolve).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("removes comment mentions when a synced comment is hard-deleted", async () => {
|
||||
const { companyId, issueId } = await createIssue();
|
||||
const commentId = randomUUID();
|
||||
await db.insert(issueComments).values({
|
||||
id: commentId,
|
||||
companyId,
|
||||
issueId,
|
||||
authorType: "user",
|
||||
authorUserId: "local-board",
|
||||
body: "See https://github.com/acme/app/issues/88",
|
||||
});
|
||||
const svc = externalObjectService(db, { github: false });
|
||||
|
||||
await svc.syncComment(commentId);
|
||||
expect(await db.select().from(externalObjectMentions)).toHaveLength(1);
|
||||
|
||||
await db.delete(issueComments).where(eq(issueComments.id, commentId));
|
||||
await svc.syncComment(commentId);
|
||||
|
||||
expect(await db.select().from(externalObjectMentions)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("skips terminal objects when refreshing due objects", async () => {
|
||||
const { companyId, issueId } = await createIssue();
|
||||
const resolve = vi.fn(async () => ({
|
||||
ok: true as const,
|
||||
snapshot: {
|
||||
statusCategory: "closed" as const,
|
||||
statusTone: "muted" as const,
|
||||
statusKey: "closed",
|
||||
statusLabel: "Closed",
|
||||
isTerminal: true,
|
||||
ttlSeconds: 1,
|
||||
},
|
||||
}));
|
||||
const resolver: ExternalObjectResolver = {
|
||||
providerKey: "url",
|
||||
objectType: "link",
|
||||
resolve,
|
||||
};
|
||||
const svc = externalObjectService(db, { resolvers: [resolver], github: false });
|
||||
await svc.syncIssue(issueId);
|
||||
const object = await db.select().from(externalObjects).then((rows) => rows[0]!);
|
||||
|
||||
await svc.refreshObject(object.id, { companyId, force: true });
|
||||
await db
|
||||
.update(externalObjects)
|
||||
.set({ nextRefreshAt: new Date(0) })
|
||||
.where(eq(externalObjects.id, object.id));
|
||||
|
||||
const refreshed = await svc.refreshDueObjects(companyId);
|
||||
|
||||
expect(refreshed).toEqual([]);
|
||||
expect(resolve).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("keeps external object identities company-scoped for duplicate urls", async () => {
|
||||
const first = await createIssue();
|
||||
const second = await createIssue();
|
||||
const svc = externalObjectService(db);
|
||||
|
||||
await svc.syncIssue(first.issueId);
|
||||
await svc.syncIssue(second.issueId);
|
||||
|
||||
const objectRows = await db.select().from(externalObjects);
|
||||
expect(objectRows).toHaveLength(2);
|
||||
expect(new Set(objectRows.map((row) => row.companyId))).toEqual(new Set([first.companyId, second.companyId]));
|
||||
expect(new Set(objectRows.map((row) => row.canonicalIdentityHash)).size).toBe(1);
|
||||
});
|
||||
|
||||
it("uses a mock plugin provider to detect and resolve non-GitHub objects", async () => {
|
||||
const { companyId, issueId } = await createIssue();
|
||||
await db
|
||||
.update(issues)
|
||||
.set({
|
||||
description: "Track https://mock.example/tickets/123?secret=drop",
|
||||
})
|
||||
.where(eq(issues.id, issueId));
|
||||
|
||||
const manifest: PaperclipPluginManifestV1 = {
|
||||
id: "paperclip.mock-object-provider",
|
||||
apiVersion: 1,
|
||||
version: "1.0.0",
|
||||
displayName: "Mock Object Provider",
|
||||
description: "Detects mock tracker tickets",
|
||||
author: "Paperclip",
|
||||
categories: ["connector"],
|
||||
capabilities: ["external.objects.detect", "external.objects.read"],
|
||||
entrypoints: { worker: "dist/worker.js" },
|
||||
objectReferences: [
|
||||
{
|
||||
providerKey: "mocktracker",
|
||||
displayName: "Mock Tracker",
|
||||
objectTypes: ["ticket"],
|
||||
urlPatterns: ["https://mock.example/tickets/:id"],
|
||||
},
|
||||
],
|
||||
};
|
||||
const [plugin] = await db.insert(plugins).values({
|
||||
pluginKey: manifest.id,
|
||||
packageName: "@paperclip/mock-object-provider",
|
||||
version: manifest.version,
|
||||
apiVersion: 1,
|
||||
categories: manifest.categories,
|
||||
manifestJson: manifest,
|
||||
status: "ready",
|
||||
installOrder: 1,
|
||||
}).returning();
|
||||
|
||||
const workerManager = {
|
||||
call: vi.fn(async (pluginId: string, method: string, params: any) => {
|
||||
expect(pluginId).toBe(plugin!.id);
|
||||
if (method === "detectExternalObjects") {
|
||||
return {
|
||||
detections: params.urls.map((url: any) => ({
|
||||
urlIdentityHash: url.canonicalIdentityHash,
|
||||
providerKey: "mocktracker",
|
||||
objectType: "ticket",
|
||||
externalId: "MOCK-123",
|
||||
displayKey: "Mock Ticket",
|
||||
iconKey: "circle-dot",
|
||||
displayTitle: "Mock ticket 123",
|
||||
confidence: "exact",
|
||||
})),
|
||||
};
|
||||
}
|
||||
if (method === "resolveExternalObject") {
|
||||
return {
|
||||
ok: true,
|
||||
snapshot: {
|
||||
displayKey: "Mock Ticket",
|
||||
iconKey: "circle-dot",
|
||||
displayTitle: `Resolved ${params.externalId}`,
|
||||
statusKey: "ready",
|
||||
statusLabel: "Ready",
|
||||
statusIconKey: "check-circle",
|
||||
statusCategory: "succeeded",
|
||||
statusTone: "success",
|
||||
ttlSeconds: 300,
|
||||
},
|
||||
};
|
||||
}
|
||||
throw new Error(`unexpected method ${method}`);
|
||||
}),
|
||||
} as unknown as PluginWorkerManager;
|
||||
|
||||
const svc = externalObjectService(db, { pluginWorkerManager: workerManager });
|
||||
await svc.syncIssue(issueId);
|
||||
|
||||
const object = await db.select().from(externalObjects).then((rows) => rows[0]!);
|
||||
expect(object).toMatchObject({
|
||||
companyId,
|
||||
providerKey: "mocktracker",
|
||||
objectType: "ticket",
|
||||
externalId: "MOCK-123",
|
||||
displayKey: "Mock Ticket",
|
||||
iconKey: "circle-dot",
|
||||
pluginId: plugin!.id,
|
||||
sanitizedCanonicalUrl: "https://mock.example/tickets/123",
|
||||
});
|
||||
expect(JSON.stringify(object)).not.toContain("secret");
|
||||
|
||||
const refreshed = await svc.refreshObject(object.id, { companyId, force: true });
|
||||
expect(refreshed.object).toMatchObject({
|
||||
displayKey: "Mock Ticket",
|
||||
iconKey: "circle-dot",
|
||||
displayTitle: "Resolved MOCK-123",
|
||||
statusIconKey: "check-circle",
|
||||
statusCategory: "succeeded",
|
||||
statusTone: "success",
|
||||
liveness: "fresh",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -22,13 +22,25 @@ import {
|
||||
issues,
|
||||
} from "@paperclipai/db";
|
||||
import { feedbackService } from "../services/feedback.ts";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.ts";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.ts";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping embedded Postgres feedback service tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
async function closeDbClient(db: ReturnType<typeof createDb> | undefined) {
|
||||
await db?.$client?.end?.({ timeout: 0 });
|
||||
}
|
||||
|
||||
describe("feedbackService.saveIssueVote", () => {
|
||||
describeEmbeddedPostgres("feedbackService.saveIssueVote", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let svc!: ReturnType<typeof feedbackService>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
@@ -15,7 +15,19 @@ import {
|
||||
import { runningProcesses } from "../adapters/index.js";
|
||||
import { heartbeatService } from "../services/heartbeat.ts";
|
||||
import { SUCCESSFUL_RUN_HANDOFF_REQUIRED_NOTICE_BODY } from "../services/recovery/index.ts";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.ts";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.ts";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping embedded Postgres heartbeat comment wake batching tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
async function waitFor(condition: () => boolean | Promise<boolean>, timeoutMs = 10_000, intervalMs = 50) {
|
||||
const startedAt = Date.now();
|
||||
@@ -147,7 +159,7 @@ async function createControlledGatewayServer() {
|
||||
};
|
||||
}
|
||||
|
||||
describe("heartbeat comment wake batching", () => {
|
||||
describeEmbeddedPostgres("heartbeat comment wake batching", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
|
||||
@@ -100,6 +100,7 @@ describe("instance settings routes", () => {
|
||||
enableExperimentalFileViewer: false,
|
||||
enableTaskWatchdogs: false,
|
||||
enableCloudSync: false,
|
||||
enableExternalObjects: false,
|
||||
autoRestartDevServerWhenIdle: false,
|
||||
enableIssueGraphLivenessAutoRecovery: true,
|
||||
issueGraphLivenessAutoRecoveryLookbackHours: 24,
|
||||
@@ -142,6 +143,7 @@ describe("instance settings routes", () => {
|
||||
enableExperimentalFileViewer: true,
|
||||
enableTaskWatchdogs: true,
|
||||
enableCloudSync: true,
|
||||
enableExternalObjects: false,
|
||||
autoRestartDevServerWhenIdle: false,
|
||||
enableIssueGraphLivenessAutoRecovery: true,
|
||||
issueGraphLivenessAutoRecoveryLookbackHours: 24,
|
||||
@@ -194,6 +196,7 @@ describe("instance settings routes", () => {
|
||||
enableExperimentalFileViewer: false,
|
||||
enableTaskWatchdogs: false,
|
||||
enableCloudSync: false,
|
||||
enableExternalObjects: false,
|
||||
autoRestartDevServerWhenIdle: false,
|
||||
enableIssueGraphLivenessAutoRecovery: true,
|
||||
issueGraphLivenessAutoRecoveryLookbackHours: 24,
|
||||
@@ -271,6 +274,24 @@ describe("instance settings routes", () => {
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("allows local board users to update external object detection", async () => {
|
||||
const app = await createApp({
|
||||
type: "board",
|
||||
userId: "local-board",
|
||||
source: "local_implicit",
|
||||
isInstanceAdmin: true,
|
||||
});
|
||||
|
||||
await request(app)
|
||||
.patch("/api/instance/settings/experimental")
|
||||
.send({ enableExternalObjects: true })
|
||||
.expect(200);
|
||||
|
||||
expect(mockInstanceSettingsService.updateExperimental).toHaveBeenCalledWith({
|
||||
enableExternalObjects: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("allows local board users to update issue graph liveness auto-recovery", async () => {
|
||||
const app = await createApp({
|
||||
type: "board",
|
||||
|
||||
@@ -19,6 +19,7 @@ describe("instance settings service", () => {
|
||||
enableIsolatedWorkspaces: true,
|
||||
enableStreamlinedLeftNavigation: true,
|
||||
enableConferenceRoomChat: false,
|
||||
enableExternalObjects: false,
|
||||
enableIssuePlanDecompositions: true,
|
||||
enableExperimentalFileViewer: true,
|
||||
enableTaskWatchdogs: true,
|
||||
|
||||
@@ -104,6 +104,34 @@ const mockHeartbeatService = vi.hoisted(() => ({
|
||||
getActiveRunForAgent: vi.fn(async () => null),
|
||||
cancelRun: vi.fn(async () => null),
|
||||
}));
|
||||
const mockExternalObjectService = vi.hoisted(() => ({
|
||||
getIssueSummaries: vi.fn(async () => new Map()),
|
||||
getIssueSummary: vi.fn(async () => ({
|
||||
authRequiredCount: 0,
|
||||
byLiveness: {},
|
||||
byStatusCategory: {},
|
||||
highestSeverity: "muted",
|
||||
objects: [],
|
||||
staleCount: 0,
|
||||
total: 0,
|
||||
unreachableCount: 0,
|
||||
})),
|
||||
getProjectSummary: vi.fn(async () => ({
|
||||
authRequiredCount: 0,
|
||||
byLiveness: {},
|
||||
byStatusCategory: {},
|
||||
highestSeverity: "muted",
|
||||
objects: [],
|
||||
staleCount: 0,
|
||||
total: 0,
|
||||
unreachableCount: 0,
|
||||
})),
|
||||
listForIssue: vi.fn(async () => []),
|
||||
refreshIssueObjects: vi.fn(async () => []),
|
||||
syncCommentSafely: vi.fn(async () => undefined),
|
||||
syncDocumentSafely: vi.fn(async () => undefined),
|
||||
syncIssueSafely: vi.fn(async () => undefined),
|
||||
}));
|
||||
|
||||
function registerRouteMocks() {
|
||||
vi.doMock("@paperclipai/shared/telemetry", () => ({
|
||||
@@ -136,6 +164,10 @@ function registerRouteMocks() {
|
||||
workProductService: () => mockWorkProductService,
|
||||
}));
|
||||
|
||||
vi.doMock("../services/external-objects.js", () => ({
|
||||
externalObjectService: () => mockExternalObjectService,
|
||||
}));
|
||||
|
||||
vi.doMock("../services/activity-log.js", () => ({
|
||||
logActivity: vi.fn(async () => undefined),
|
||||
}));
|
||||
@@ -329,6 +361,7 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
vi.doUnmock("../services/activity-log.js");
|
||||
vi.doUnmock("../services/agents.js");
|
||||
vi.doUnmock("../services/documents.js");
|
||||
vi.doUnmock("../services/external-objects.js");
|
||||
vi.doUnmock("../services/index.js");
|
||||
vi.doUnmock("../services/issues.js");
|
||||
vi.doUnmock("../services/work-products.js");
|
||||
@@ -464,6 +497,14 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
mockIssueService.findMentionedAgents.mockReset();
|
||||
mockDocumentService.upsertIssueDocument.mockReset();
|
||||
mockWorkProductService.createForIssue.mockReset();
|
||||
mockExternalObjectService.getIssueSummaries.mockClear();
|
||||
mockExternalObjectService.getIssueSummary.mockClear();
|
||||
mockExternalObjectService.getProjectSummary.mockClear();
|
||||
mockExternalObjectService.listForIssue.mockClear();
|
||||
mockExternalObjectService.refreshIssueObjects.mockClear();
|
||||
mockExternalObjectService.syncCommentSafely.mockClear();
|
||||
mockExternalObjectService.syncDocumentSafely.mockClear();
|
||||
mockExternalObjectService.syncIssueSafely.mockClear();
|
||||
mockWorkProductService.getById.mockReset();
|
||||
mockWorkProductService.remove.mockReset();
|
||||
mockWorkProductService.update.mockReset();
|
||||
|
||||
@@ -57,6 +57,24 @@ const mockIssueReferenceService = vi.hoisted(() => ({
|
||||
syncDocument: vi.fn(async () => undefined),
|
||||
syncIssue: vi.fn(async () => undefined),
|
||||
}));
|
||||
const mockExternalObjectService = vi.hoisted(() => ({
|
||||
getIssueSummaries: vi.fn(async () => ({ summaries: {} })),
|
||||
getIssueSummary: vi.fn(async () => ({
|
||||
authRequiredCount: 0,
|
||||
byLiveness: {},
|
||||
byStatusCategory: {},
|
||||
highestSeverity: "muted",
|
||||
objects: [],
|
||||
staleCount: 0,
|
||||
total: 0,
|
||||
unreachableCount: 0,
|
||||
})),
|
||||
listForIssue: vi.fn(async () => []),
|
||||
refreshIssueObjects: vi.fn(async () => []),
|
||||
syncCommentSafely: vi.fn(async () => undefined),
|
||||
syncDocumentSafely: vi.fn(async () => undefined),
|
||||
syncIssueSafely: vi.fn(async () => undefined),
|
||||
}));
|
||||
|
||||
function registerModuleMocks() {
|
||||
vi.doMock("@paperclipai/shared/telemetry", () => ({
|
||||
@@ -92,6 +110,10 @@ function registerModuleMocks() {
|
||||
issueService: () => mockIssueService,
|
||||
}));
|
||||
|
||||
vi.doMock("../services/external-objects.js", () => ({
|
||||
externalObjectService: () => mockExternalObjectService,
|
||||
}));
|
||||
|
||||
vi.doMock("../services/index.js", () => ({
|
||||
companyService: () => ({
|
||||
getById: vi.fn(async () => ({ id: "company-1", attachmentMaxBytes: 10 * 1024 * 1024 })),
|
||||
@@ -181,6 +203,7 @@ describe.sequential("issue comment cancel routes", () => {
|
||||
vi.doUnmock("../telemetry.js");
|
||||
vi.doUnmock("../services/access.js");
|
||||
vi.doUnmock("../services/activity-log.js");
|
||||
vi.doUnmock("../services/external-objects.js");
|
||||
vi.doUnmock("../services/feedback.js");
|
||||
vi.doUnmock("../services/heartbeat.js");
|
||||
vi.doUnmock("../services/index.js");
|
||||
@@ -240,6 +263,7 @@ describe.sequential("issue comment cancel routes", () => {
|
||||
});
|
||||
mockIssueReferenceService.deleteCommentSource.mockResolvedValue(undefined);
|
||||
mockIssueReferenceService.syncComment.mockResolvedValue(undefined);
|
||||
mockExternalObjectService.syncCommentSafely.mockResolvedValue(undefined);
|
||||
});
|
||||
|
||||
it("cancels a queued comment from its author and restores the deleted body", async () => {
|
||||
@@ -352,6 +376,7 @@ describe.sequential("issue comment cancel routes", () => {
|
||||
expect.objectContaining({ afterTombstone: expect.any(Function) }),
|
||||
);
|
||||
expect(mockIssueReferenceService.syncComment).toHaveBeenCalledWith("comment-1", "tx");
|
||||
expect(mockExternalObjectService.syncCommentSafely).toHaveBeenCalledWith("comment-1", "tx");
|
||||
expect(mockDocumentAnnotationService.cleanupForIssueCommentDeletion).toHaveBeenCalledWith(
|
||||
"11111111-1111-4111-8111-111111111111",
|
||||
"comment-1",
|
||||
@@ -362,6 +387,7 @@ describe.sequential("issue comment cancel routes", () => {
|
||||
"tx",
|
||||
);
|
||||
expect(mockIssueReferenceService.deleteCommentSource).toHaveBeenCalledWith("annotation-comment-1", "tx");
|
||||
expect(mockExternalObjectService.syncCommentSafely).toHaveBeenCalledWith("annotation-comment-1", "tx");
|
||||
const deletedActivity = mockLogActivity.mock.calls.find((call) => call[1]?.action === "issue.comment_deleted")?.[1];
|
||||
expect(deletedActivity).toEqual(expect.objectContaining({
|
||||
action: "issue.comment_deleted",
|
||||
|
||||
@@ -79,6 +79,10 @@ const mockIssueRecoveryActionService = vi.hoisted(() => ({
|
||||
const mockIssueTreeControlService = vi.hoisted(() => ({
|
||||
getActivePauseHoldGate: vi.fn(async () => null),
|
||||
}));
|
||||
const mockExternalObjectService = vi.hoisted(() => ({
|
||||
syncCommentSafely: vi.fn(async () => undefined),
|
||||
syncIssueSafely: vi.fn(async () => undefined),
|
||||
}));
|
||||
|
||||
vi.mock("@paperclipai/shared/telemetry", () => ({
|
||||
trackAgentTaskCompleted: vi.fn(),
|
||||
@@ -158,6 +162,10 @@ vi.mock("../services/index.js", () => ({
|
||||
workProductService: () => ({}),
|
||||
}));
|
||||
|
||||
vi.mock("../services/external-objects.js", () => ({
|
||||
externalObjectService: () => mockExternalObjectService,
|
||||
}));
|
||||
|
||||
function createApp() {
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
@@ -253,6 +261,8 @@ describe.sequential("issue comment reopen routes", () => {
|
||||
mockRoutineService.syncRunStatusForIssue.mockReset();
|
||||
mockIssueRecoveryActionService.getActiveForIssue.mockReset();
|
||||
mockIssueTreeControlService.getActivePauseHoldGate.mockReset();
|
||||
mockExternalObjectService.syncCommentSafely.mockReset();
|
||||
mockExternalObjectService.syncIssueSafely.mockReset();
|
||||
mockTxInsertValues.mockReset();
|
||||
mockTxInsert.mockReset();
|
||||
mockDbSelect.mockReset();
|
||||
@@ -276,6 +286,8 @@ describe.sequential("issue comment reopen routes", () => {
|
||||
mockHeartbeatService.getRun.mockResolvedValue(null);
|
||||
mockHeartbeatService.getActiveRunForAgent.mockResolvedValue(null);
|
||||
mockHeartbeatService.cancelRun.mockResolvedValue(null);
|
||||
mockExternalObjectService.syncCommentSafely.mockResolvedValue(undefined);
|
||||
mockExternalObjectService.syncIssueSafely.mockResolvedValue(undefined);
|
||||
mockLogActivity.mockResolvedValue(undefined);
|
||||
mockFeedbackService.listIssueVotesForUser.mockResolvedValue([]);
|
||||
mockFeedbackService.saveIssueVote.mockResolvedValue({
|
||||
|
||||
@@ -159,6 +159,203 @@ describe("plugin environment driver seam", () => {
|
||||
});
|
||||
});
|
||||
|
||||
const objectReferenceManifest: PaperclipPluginManifestV1 = {
|
||||
id: "test.external-object-provider",
|
||||
apiVersion: 1,
|
||||
version: "1.0.0",
|
||||
displayName: "External Object Provider",
|
||||
description: "Test external object provider plugin",
|
||||
author: "Paperclip",
|
||||
categories: ["connector"],
|
||||
capabilities: ["external.objects.detect", "external.objects.read"],
|
||||
entrypoints: { worker: "dist/worker.js" },
|
||||
objectReferences: [
|
||||
{
|
||||
providerKey: "mocktracker",
|
||||
displayName: "Mock Tracker",
|
||||
objectTypes: ["ticket"],
|
||||
urlPatterns: ["https://mock.example/tickets/:id"],
|
||||
refreshPolicy: { defaultTtlSeconds: 300, staleAfterSeconds: 1800 },
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
describe("plugin external object provider seam", () => {
|
||||
it("validates provider manifest declarations and capabilities", () => {
|
||||
expect(pluginManifestV1Schema.safeParse(objectReferenceManifest).success).toBe(true);
|
||||
|
||||
const missingCapability = pluginManifestV1Schema.safeParse({
|
||||
...objectReferenceManifest,
|
||||
capabilities: ["external.objects.detect"],
|
||||
});
|
||||
expect(missingCapability.success).toBe(false);
|
||||
expect(JSON.stringify(missingCapability.error?.issues)).toContain("external.objects.read");
|
||||
|
||||
const duplicateProvider = pluginManifestV1Schema.safeParse({
|
||||
...objectReferenceManifest,
|
||||
objectReferences: [
|
||||
objectReferenceManifest.objectReferences![0],
|
||||
{ ...objectReferenceManifest.objectReferences![0], displayName: "Duplicate" },
|
||||
],
|
||||
});
|
||||
expect(duplicateProvider.success).toBe(false);
|
||||
expect(JSON.stringify(duplicateProvider.error?.issues)).toContain(
|
||||
"Duplicate object reference provider keys",
|
||||
);
|
||||
});
|
||||
|
||||
it("enforces provider capability requirements", () => {
|
||||
const validator = pluginCapabilityValidator();
|
||||
expect(validator.getRequiredCapabilities("external.objects.detect")).toEqual([
|
||||
"external.objects.detect",
|
||||
]);
|
||||
expect(validator.getRequiredCapabilities("external.objects.read")).toEqual([
|
||||
"external.objects.read",
|
||||
]);
|
||||
expect(validator.checkOperation(objectReferenceManifest, "external.objects.read").allowed).toBe(true);
|
||||
|
||||
const withoutCapability = {
|
||||
...objectReferenceManifest,
|
||||
capabilities: ["external.objects.detect"],
|
||||
} satisfies PaperclipPluginManifestV1;
|
||||
|
||||
expect(validator.checkOperation(withoutCapability, "external.objects.read")).toMatchObject({
|
||||
allowed: false,
|
||||
missing: ["external.objects.read"],
|
||||
});
|
||||
expect(validator.validateManifestCapabilities(withoutCapability)).toMatchObject({
|
||||
allowed: false,
|
||||
missing: ["external.objects.read"],
|
||||
});
|
||||
});
|
||||
|
||||
it("dispatches provider detection and resolution worker hooks", async () => {
|
||||
const plugin = definePlugin({
|
||||
async setup() {},
|
||||
async onDetectExternalObjects(params) {
|
||||
return {
|
||||
detections: params.urls.map((url) => ({
|
||||
urlIdentityHash: url.canonicalIdentityHash,
|
||||
providerKey: "mocktracker",
|
||||
objectType: "ticket",
|
||||
externalId: "MOCK-123",
|
||||
displayTitle: "Mock ticket",
|
||||
confidence: "exact",
|
||||
})),
|
||||
};
|
||||
},
|
||||
async onResolveExternalObject(params) {
|
||||
return {
|
||||
ok: true,
|
||||
snapshot: {
|
||||
displayTitle: `Resolved ${params.externalId}`,
|
||||
statusKey: "ready",
|
||||
statusLabel: "Ready",
|
||||
statusCategory: "succeeded",
|
||||
statusTone: "success",
|
||||
isTerminal: true,
|
||||
ttlSeconds: 600,
|
||||
},
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
const stdin = new PassThrough();
|
||||
const stdout = new PassThrough();
|
||||
const host = startWorkerRpcHost({ plugin, stdin, stdout });
|
||||
const responses: unknown[] = [];
|
||||
stdout.on("data", (chunk) => {
|
||||
const lines = String(chunk).split("\n").filter(Boolean);
|
||||
for (const line of lines) {
|
||||
responses.push(parseMessage(line));
|
||||
}
|
||||
});
|
||||
|
||||
stdin.write(serializeMessage(createRequest("initialize", {
|
||||
manifest: objectReferenceManifest,
|
||||
config: {},
|
||||
instanceInfo: { instanceId: "instance-1", hostVersion: "1.0.0" },
|
||||
apiVersion: 1,
|
||||
}, 1)));
|
||||
await waitForResponses(responses, 1);
|
||||
|
||||
const initializeResponse = responses[0];
|
||||
expect(isJsonRpcSuccessResponse(initializeResponse)).toBe(true);
|
||||
if (!isJsonRpcSuccessResponse(initializeResponse)) return;
|
||||
expect(initializeResponse.result.supportedMethods).toContain("detectExternalObjects");
|
||||
expect(initializeResponse.result.supportedMethods).toContain("resolveExternalObject");
|
||||
|
||||
stdin.write(serializeMessage(createRequest("detectExternalObjects", {
|
||||
companyId: "company-1",
|
||||
urls: [{
|
||||
sanitizedCanonicalUrl: "https://mock.example/tickets/123",
|
||||
sanitizedDisplayUrl: "https://mock.example/tickets/123",
|
||||
canonicalIdentityHash: "hash-123",
|
||||
canonicalIdentity: { scheme: "https", host: "mock.example", path: "/tickets/123" },
|
||||
redactedMatchedText: "https://mock.example/tickets/123",
|
||||
}],
|
||||
sourceContext: {
|
||||
companyId: "company-1",
|
||||
sourceIssueId: "issue-1",
|
||||
sourceKind: "description",
|
||||
sourceRecordId: null,
|
||||
documentKey: null,
|
||||
propertyKey: null,
|
||||
},
|
||||
}, 2)));
|
||||
await waitForResponses(responses, 2);
|
||||
|
||||
const detectResponse = responses[1];
|
||||
expect(isJsonRpcSuccessResponse(detectResponse)).toBe(true);
|
||||
if (!isJsonRpcSuccessResponse(detectResponse)) return;
|
||||
expect(detectResponse.result.detections[0]).toMatchObject({
|
||||
providerKey: "mocktracker",
|
||||
objectType: "ticket",
|
||||
externalId: "MOCK-123",
|
||||
});
|
||||
|
||||
stdin.write(serializeMessage(createRequest("resolveExternalObject", {
|
||||
companyId: "company-1",
|
||||
providerKey: "mocktracker",
|
||||
objectType: "ticket",
|
||||
externalId: "MOCK-123",
|
||||
object: {
|
||||
id: "object-1",
|
||||
companyId: "company-1",
|
||||
providerKey: "mocktracker",
|
||||
objectType: "ticket",
|
||||
externalId: "MOCK-123",
|
||||
sanitizedCanonicalUrl: "https://mock.example/tickets/123",
|
||||
canonicalIdentityHash: "hash-123",
|
||||
displayTitle: "Mock ticket",
|
||||
statusKey: null,
|
||||
statusLabel: null,
|
||||
statusCategory: "unknown",
|
||||
statusTone: "neutral",
|
||||
liveness: "unknown",
|
||||
isTerminal: false,
|
||||
data: {},
|
||||
remoteVersion: null,
|
||||
etag: null,
|
||||
},
|
||||
}, 3)));
|
||||
await waitForResponses(responses, 3);
|
||||
|
||||
const resolveResponse = responses[2];
|
||||
expect(isJsonRpcSuccessResponse(resolveResponse)).toBe(true);
|
||||
if (!isJsonRpcSuccessResponse(resolveResponse)) return;
|
||||
expect(resolveResponse.result).toMatchObject({
|
||||
ok: true,
|
||||
snapshot: {
|
||||
statusCategory: "succeeded",
|
||||
statusTone: "success",
|
||||
},
|
||||
});
|
||||
|
||||
host.stop();
|
||||
});
|
||||
});
|
||||
|
||||
async function waitForResponses(responses: unknown[], count: number): Promise<void> {
|
||||
const deadline = Date.now() + 1_000;
|
||||
while (responses.length < count && Date.now() < deadline) {
|
||||
|
||||
@@ -1536,6 +1536,97 @@ describe("realizeExecutionWorkspace", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("regenerates stale worktree config that points at another host", async () => {
|
||||
const tempRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-worktree-stale-config-"));
|
||||
const baseRoot = path.join(tempRoot, "base");
|
||||
const worktreeRoot = path.join(tempRoot, "worktree");
|
||||
const fakeBin = path.join(tempRoot, "bin");
|
||||
const fakePnpmPath = path.join(fakeBin, "pnpm");
|
||||
const scriptPath = path.join(worktreeRoot, "provision-worktree.sh");
|
||||
const paperclipDir = path.join(worktreeRoot, ".paperclip");
|
||||
|
||||
try {
|
||||
await fs.mkdir(baseRoot, { recursive: true });
|
||||
await fs.mkdir(paperclipDir, { recursive: true });
|
||||
await fs.mkdir(fakeBin, { recursive: true });
|
||||
await fs.copyFile(provisionWorktreeScriptPath, scriptPath);
|
||||
await fs.chmod(scriptPath, 0o755);
|
||||
await fs.writeFile(
|
||||
path.join(paperclipDir, "config.json"),
|
||||
JSON.stringify({
|
||||
database: {
|
||||
mode: "embedded-postgres",
|
||||
embeddedPostgresDataDir: "/Users/example/.paperclip-worktrees/instances/stale/db",
|
||||
},
|
||||
logging: {
|
||||
mode: "file",
|
||||
logDir: "/Users/example/.paperclip-worktrees/instances/stale/logs",
|
||||
},
|
||||
storage: {
|
||||
provider: "local_disk",
|
||||
localDisk: {
|
||||
baseDir: "/Users/example/.paperclip-worktrees/instances/stale/data/storage",
|
||||
},
|
||||
},
|
||||
secrets: {
|
||||
provider: "local_encrypted",
|
||||
localEncrypted: {
|
||||
keyFilePath: "/Users/example/.paperclip-worktrees/instances/stale/secrets/master.key",
|
||||
},
|
||||
},
|
||||
}),
|
||||
"utf8",
|
||||
);
|
||||
await fs.writeFile(
|
||||
path.join(paperclipDir, ".env"),
|
||||
[
|
||||
"PAPERCLIP_HOME=/Users/example/.paperclip-worktrees",
|
||||
"PAPERCLIP_INSTANCE_ID=stale",
|
||||
`PAPERCLIP_CONFIG=/Users/example/paperclip/${path.basename(worktreeRoot)}/.paperclip/config.json`,
|
||||
"",
|
||||
].join("\n"),
|
||||
"utf8",
|
||||
);
|
||||
await fs.writeFile(
|
||||
fakePnpmPath,
|
||||
[
|
||||
"#!/bin/sh",
|
||||
"if [ \"$1\" = \"paperclipai\" ] && [ \"$2\" = \"--help\" ]; then",
|
||||
" exit 0",
|
||||
"fi",
|
||||
"if [ \"$1\" = \"paperclipai\" ] && [ \"$2\" = \"worktree\" ] && [ \"$3\" = \"init\" ]; then",
|
||||
" mkdir -p \"$PWD/.paperclip\"",
|
||||
" printf '%s\\n' '{\"database\":{\"embeddedPostgresDataDir\":\"'$PWD'/.paperclip/runtime/db\"}}' > \"$PWD/.paperclip/config.json\"",
|
||||
" printf '%s\\n' \"PAPERCLIP_HOME=$PWD/.paperclip/runtime\" \"PAPERCLIP_INSTANCE_ID=healthy\" \"PAPERCLIP_CONFIG=$PWD/.paperclip/config.json\" > \"$PWD/.paperclip/.env\"",
|
||||
" exit 0",
|
||||
"fi",
|
||||
"exit 0",
|
||||
"",
|
||||
].join("\n"),
|
||||
"utf8",
|
||||
);
|
||||
await fs.chmod(fakePnpmPath, 0o755);
|
||||
|
||||
const result = await execFileAsync(scriptPath, [], {
|
||||
cwd: worktreeRoot,
|
||||
env: {
|
||||
...process.env,
|
||||
PATH: `${fakeBin}:${process.env.PATH ?? ""}`,
|
||||
PAPERCLIP_WORKSPACE_BASE_CWD: baseRoot,
|
||||
PAPERCLIP_WORKSPACE_CWD: worktreeRoot,
|
||||
},
|
||||
});
|
||||
|
||||
expect(result.stderr).toContain("Existing isolated Paperclip worktree config is stale for this host; regenerating.");
|
||||
await expect(fs.readFile(path.join(paperclipDir, ".env"), "utf8")).resolves.toContain(
|
||||
`PAPERCLIP_CONFIG=${worktreeRoot}/.paperclip/config.json`,
|
||||
);
|
||||
await expect(fs.readFile(path.join(paperclipDir, "config.json"), "utf8")).resolves.toContain(worktreeRoot);
|
||||
} finally {
|
||||
await fs.rm(tempRoot, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("retries worktree-local pnpm install without a frozen lockfile when the lockfile is outdated", async () => {
|
||||
const tempRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-worktree-outdated-lockfile-"));
|
||||
const baseRoot = path.join(tempRoot, "base");
|
||||
|
||||
@@ -148,11 +148,18 @@ import {
|
||||
resolveCoreTrustPreset,
|
||||
type TrustPresetResolution,
|
||||
} from "../services/trust-preset-resolver.js";
|
||||
import { externalObjectService } from "../services/external-objects.js";
|
||||
|
||||
const MAX_ISSUE_COMMENT_LIMIT = 500;
|
||||
const updateIssueRouteSchema = updateIssueSchema.extend({
|
||||
interrupt: z.boolean().optional(),
|
||||
});
|
||||
const refreshExternalObjectsSchema = z.object({
|
||||
objectIds: z.array(z.string().uuid()).max(50).optional(),
|
||||
}).strict();
|
||||
const externalObjectSummariesSchema = z.object({
|
||||
issueIds: z.array(z.string().uuid()).max(1000),
|
||||
}).strict();
|
||||
|
||||
const promoteLowTrustOutputSchema = z.object({
|
||||
sourceArtifactKind: z.enum(["comment", "document", "work_product", "issue"]),
|
||||
@@ -1105,6 +1112,10 @@ export function issueRoutes(
|
||||
? heartbeat.wakeup
|
||||
: opts.taskWatchdogEnqueueWakeup ?? undefined,
|
||||
}) ?? noopTaskWatchdogService();
|
||||
const externalObjectsSvc = externalObjectService(db, {
|
||||
pluginWorkerManager: opts.pluginWorkerManager,
|
||||
enabled: async () => (await instanceSettings.getExperimental()).enableExternalObjects === true,
|
||||
});
|
||||
const routinesSvc = routineService(db, {
|
||||
pluginWorkerManager: opts.pluginWorkerManager,
|
||||
});
|
||||
@@ -3730,6 +3741,69 @@ export function issueRoutes(
|
||||
res.json(workProducts);
|
||||
});
|
||||
|
||||
router.get("/issues/:id/external-objects", async (req, res) => {
|
||||
const id = req.params.id as string;
|
||||
const issue = await svc.getById(id);
|
||||
if (!issue) {
|
||||
res.status(404).json({ error: "Issue not found" });
|
||||
return;
|
||||
}
|
||||
assertCompanyAccess(req, issue.companyId);
|
||||
const objects = await externalObjectsSvc.listForIssue(issue.id);
|
||||
res.json(objects);
|
||||
});
|
||||
|
||||
router.get("/issues/:id/external-object-summary", async (req, res) => {
|
||||
const id = req.params.id as string;
|
||||
const issue = await svc.getById(id);
|
||||
if (!issue) {
|
||||
res.status(404).json({ error: "Issue not found" });
|
||||
return;
|
||||
}
|
||||
assertCompanyAccess(req, issue.companyId);
|
||||
const summary = await externalObjectsSvc.getIssueSummary(issue.id);
|
||||
res.json(summary);
|
||||
});
|
||||
|
||||
router.post("/companies/:companyId/issues/external-object-summaries", validate(externalObjectSummariesSchema), async (req, res) => {
|
||||
const companyId = req.params.companyId as string;
|
||||
assertCompanyAccess(req, companyId);
|
||||
const summaries = await externalObjectsSvc.getIssueSummaries(companyId, req.body.issueIds);
|
||||
res.json({ summaries: Object.fromEntries(summaries) });
|
||||
});
|
||||
|
||||
router.post("/issues/:id/external-objects/refresh", validate(refreshExternalObjectsSchema), async (req, res) => {
|
||||
const id = req.params.id as string;
|
||||
const issue = await svc.getById(id);
|
||||
if (!issue) {
|
||||
res.status(404).json({ error: "Issue not found" });
|
||||
return;
|
||||
}
|
||||
assertCompanyAccess(req, issue.companyId);
|
||||
if (!(await assertAgentIssueMutationAllowed(req, res, issue))) return;
|
||||
const actor = getActorInfo(req);
|
||||
const results = await externalObjectsSvc.refreshIssueObjects(issue.id, {
|
||||
companyId: issue.companyId,
|
||||
objectIds: req.body.objectIds,
|
||||
actor,
|
||||
});
|
||||
await logActivity(db, {
|
||||
companyId: issue.companyId,
|
||||
actorType: actor.actorType,
|
||||
actorId: actor.actorId,
|
||||
agentId: actor.agentId,
|
||||
runId: actor.runId,
|
||||
action: "external_object.refresh_requested",
|
||||
entityType: "issue",
|
||||
entityId: issue.id,
|
||||
details: {
|
||||
issueId: issue.id,
|
||||
objectIds: results.map((result) => result.object.id),
|
||||
},
|
||||
});
|
||||
res.json({ refreshed: results });
|
||||
});
|
||||
|
||||
router.get("/issues/:id/documents", async (req, res) => {
|
||||
const id = req.params.id as string;
|
||||
const issue = await svc.getById(id);
|
||||
@@ -4016,6 +4090,7 @@ export function issueRoutes(
|
||||
const redirectedFromLockedDocument =
|
||||
"redirectedFromLockedDocument" in result ? result.redirectedFromLockedDocument : null;
|
||||
await issueReferencesSvc.syncDocument(doc.id);
|
||||
await externalObjectsSvc.syncDocumentSafely(doc.id);
|
||||
const referenceSummaryAfter = await issueReferencesSvc.listIssueReferenceSummary(issue.id);
|
||||
const referenceDiff = issueReferencesSvc.diffIssueReferenceSummary(referenceSummaryBefore, referenceSummaryAfter);
|
||||
const remappedAnnotations = result.created
|
||||
@@ -4245,6 +4320,7 @@ export function issueRoutes(
|
||||
});
|
||||
await issueReferencesSvc.syncDocument(result.document.id);
|
||||
const referenceSummaryAfter = await issueReferencesSvc.listIssueReferenceSummary(issue.id);
|
||||
await externalObjectsSvc.syncDocumentSafely(result.document.id);
|
||||
const referenceDiff = issueReferencesSvc.diffIssueReferenceSummary(referenceSummaryBefore, referenceSummaryAfter);
|
||||
const remappedAnnotations = await documentAnnotationsSvc.remapOpenThreadsForDocument({
|
||||
issueId: issue.id,
|
||||
@@ -4358,6 +4434,7 @@ export function issueRoutes(
|
||||
}
|
||||
await issueReferencesSvc.deleteDocumentSource(removed.id);
|
||||
const referenceSummaryAfter = await issueReferencesSvc.listIssueReferenceSummary(issue.id);
|
||||
if (removed) await externalObjectsSvc.syncDocumentSafely(removed.id);
|
||||
const referenceDiff = issueReferencesSvc.diffIssueReferenceSummary(referenceSummaryBefore, referenceSummaryAfter);
|
||||
const actor = getActorInfo(req);
|
||||
await logActivity(db, {
|
||||
@@ -5025,6 +5102,7 @@ export function issueRoutes(
|
||||
watchdogActorRunId: actor.runId,
|
||||
});
|
||||
await issueReferencesSvc.syncIssue(issue.id);
|
||||
await externalObjectsSvc.syncIssueSafely(issue.id);
|
||||
const referenceSummary = await issueReferencesSvc.listIssueReferenceSummary(issue.id);
|
||||
const referenceDiff = issueReferencesSvc.diffIssueReferenceSummary(
|
||||
issueReferencesSvc.emptySummary(),
|
||||
@@ -5190,6 +5268,7 @@ export function issueRoutes(
|
||||
actorUserId: actor.actorType === "user" ? actor.actorId : null,
|
||||
watchdogActorRunId: actor.runId,
|
||||
});
|
||||
await externalObjectsSvc.syncIssueSafely(issue.id);
|
||||
|
||||
await logActivity(db, {
|
||||
companyId: parent.companyId,
|
||||
@@ -5958,6 +6037,7 @@ export function issueRoutes(
|
||||
|
||||
if (titleOrDescriptionChanged) {
|
||||
await issueReferencesSvc.syncIssue(issue.id);
|
||||
await externalObjectsSvc.syncIssueSafely(issue.id);
|
||||
}
|
||||
const updateReferenceSummaryAfter = titleOrDescriptionChanged
|
||||
? await issueReferencesSvc.listIssueReferenceSummary(issue.id)
|
||||
@@ -6263,6 +6343,7 @@ export function issueRoutes(
|
||||
sourceTrust: await sourceTrustForActorWrite(issue, actor),
|
||||
});
|
||||
await issueReferencesSvc.syncComment(comment.id);
|
||||
await externalObjectsSvc.syncCommentSafely(comment.id);
|
||||
const commentReferenceSummaryAfter = await issueReferencesSvc.listIssueReferenceSummary(issue.id);
|
||||
const commentReferenceDiff = issueReferencesSvc.diffIssueReferenceSummary(
|
||||
commentReferenceSummaryBefore,
|
||||
@@ -7256,6 +7337,7 @@ export function issueRoutes(
|
||||
{
|
||||
afterTombstone: async (deletedComment, tx) => {
|
||||
await issueReferencesSvc.syncComment(deletedComment.id, tx);
|
||||
await externalObjectsSvc.syncCommentSafely(deletedComment.id, tx);
|
||||
annotationCleanup = await documentAnnotationsSvc.cleanupForIssueCommentDeletion(issue.id, deletedComment.id, {
|
||||
actorType: actor.actorType,
|
||||
actorId: actor.actorId,
|
||||
@@ -7265,7 +7347,10 @@ export function issueRoutes(
|
||||
}, tx);
|
||||
await Promise.all(
|
||||
annotationCleanup.deletedCommentIds.map((annotationCommentId) =>
|
||||
issueReferencesSvc.deleteCommentSource(annotationCommentId, tx)
|
||||
Promise.all([
|
||||
issueReferencesSvc.deleteCommentSource(annotationCommentId, tx),
|
||||
externalObjectsSvc.syncCommentSafely(annotationCommentId, tx),
|
||||
])
|
||||
),
|
||||
);
|
||||
},
|
||||
@@ -7705,6 +7790,7 @@ export function issueRoutes(
|
||||
}
|
||||
|
||||
await issueReferencesSvc.syncComment(comment.id);
|
||||
await externalObjectsSvc.syncCommentSafely(comment.id);
|
||||
const commentReferenceSummaryAfter = await issueReferencesSvc.listIssueReferenceSummary(currentIssue.id);
|
||||
const commentReferenceDiff = issueReferencesSvc.diffIssueReferenceSummary(
|
||||
commentReferenceSummaryBefore,
|
||||
|
||||
@@ -447,6 +447,14 @@ const jsonBody = (schema: z.ZodTypeAny) => ({
|
||||
|
||||
const r = responses;
|
||||
|
||||
const externalObjectSummariesBodySchema = z.object({
|
||||
issueIds: z.array(z.string().uuid()).max(1000),
|
||||
}).strict();
|
||||
|
||||
const refreshExternalObjectsBodySchema = z.object({
|
||||
objectIds: z.array(z.string().uuid()).max(50).optional(),
|
||||
}).strict();
|
||||
|
||||
function paramsSchemaFromPath(routePath: string): z.ZodObject<z.ZodRawShape> | undefined {
|
||||
const names = [...routePath.matchAll(/\{([A-Za-z0-9_]+)\}/g)].map((match) => match[1]);
|
||||
if (names.length === 0) return undefined;
|
||||
@@ -3847,6 +3855,57 @@ registry.registerPath({
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 404: r.notFound },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "get",
|
||||
path: "/api/issues/{id}/external-objects",
|
||||
tags: ["issues"],
|
||||
summary: "List external objects mentioned by an issue",
|
||||
request: { params: z.object({ id: z.string() }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "get",
|
||||
path: "/api/issues/{id}/external-object-summary",
|
||||
tags: ["issues"],
|
||||
summary: "Get external object status summary for an issue",
|
||||
request: { params: z.object({ id: z.string() }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "post",
|
||||
path: "/api/companies/{companyId}/issues/external-object-summaries",
|
||||
tags: ["issues"],
|
||||
summary: "Get external object status summaries for issues",
|
||||
request: {
|
||||
params: z.object({ companyId: z.string() }),
|
||||
body: jsonBody(externalObjectSummariesBodySchema),
|
||||
},
|
||||
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "post",
|
||||
path: "/api/issues/{id}/external-objects/refresh",
|
||||
tags: ["issues"],
|
||||
summary: "Refresh external objects mentioned by an issue",
|
||||
request: {
|
||||
params: z.object({ id: z.string() }),
|
||||
body: jsonBody(refreshExternalObjectsBodySchema),
|
||||
},
|
||||
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "get",
|
||||
path: "/api/projects/{id}/external-object-summary",
|
||||
tags: ["projects"],
|
||||
summary: "Get external object status summary for a project",
|
||||
request: { params: z.object({ id: z.string() }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
|
||||
});
|
||||
|
||||
// ─── Org chart images ─────────────────────────────────────────────────────────
|
||||
|
||||
registry.registerPath({
|
||||
|
||||
@@ -15,6 +15,8 @@ import { trackProjectCreated } from "@paperclipai/shared/telemetry";
|
||||
import { validate } from "../middleware/validate.js";
|
||||
import { accessService, projectService, logActivity, workspaceOperationService } from "../services/index.js";
|
||||
import { conflict, forbidden } from "../errors.js";
|
||||
import { externalObjectService } from "../services/external-objects.js";
|
||||
import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
import { assertCompanyAccess, getActorInfo } from "./authz.js";
|
||||
import {
|
||||
buildWorkspaceRuntimeDesiredStatePatch,
|
||||
@@ -44,6 +46,10 @@ export function projectRoutes(db: Db) {
|
||||
const access = accessService(db);
|
||||
const secretsSvc = secretService(db);
|
||||
const workspaceOperations = workspaceOperationService(db);
|
||||
const instanceSettings = instanceSettingsService(db);
|
||||
const externalObjectsSvc = externalObjectService(db, {
|
||||
enabled: async () => (await instanceSettings.getExperimental()).enableExternalObjects === true,
|
||||
});
|
||||
const strictSecretsMode = process.env.PAPERCLIP_SECRETS_STRICT_MODE === "true";
|
||||
const environmentsSvc = environmentService(db);
|
||||
|
||||
@@ -139,6 +145,18 @@ export function projectRoutes(db: Db) {
|
||||
res.json(project);
|
||||
});
|
||||
|
||||
router.get("/projects/:id/external-object-summary", async (req, res) => {
|
||||
const id = req.params.id as string;
|
||||
const project = await svc.getById(id);
|
||||
if (!project) {
|
||||
res.status(404).json({ error: "Project not found" });
|
||||
return;
|
||||
}
|
||||
assertCompanyAccess(req, project.companyId);
|
||||
const summary = await externalObjectsSvc.getProjectSummary(project.id);
|
||||
res.json(summary);
|
||||
});
|
||||
|
||||
router.post("/companies/:companyId/projects", validate(createProjectSchema), async (req, res) => {
|
||||
const companyId = req.params.companyId as string;
|
||||
assertCompanyAccess(req, companyId);
|
||||
|
||||
@@ -0,0 +1,945 @@
|
||||
import { and, asc, eq, inArray, isNull, lte, sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { documents, externalObjectMentions, externalObjects, issueComments, issueDocuments, issues, plugins } from "@paperclipai/db";
|
||||
import {
|
||||
formatExternalObjectMentionSourceLabel,
|
||||
type ExternalObjectCanonicalUrl,
|
||||
type ExternalObjectLivenessState,
|
||||
type ExternalObjectMentionConfidence,
|
||||
type ExternalObjectMentionSourceKind,
|
||||
type ExternalObjectStatusCategory,
|
||||
type ExternalObjectStatusTone,
|
||||
type PaperclipPluginManifestV1,
|
||||
} from "@paperclipai/shared";
|
||||
import { extractExternalObjectCanonicalUrls } from "@paperclipai/shared/external-objects-server";
|
||||
import type { PluginExternalObjectRecordSnapshot, PluginExternalObjectResolveResult } from "@paperclipai/plugin-sdk";
|
||||
import { notFound } from "../errors.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { logActivity, type LogActivityInput } from "./activity-log.js";
|
||||
import { createGitHubExternalObjectProvider, type GitHubExternalObjectProviderOptions } from "./github-external-object-provider.js";
|
||||
import { publishLiveEvent } from "./live-events.js";
|
||||
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
||||
|
||||
export interface ExternalObjectSourceContext {
|
||||
companyId: string;
|
||||
sourceIssueId: string;
|
||||
sourceKind: ExternalObjectMentionSourceKind;
|
||||
sourceRecordId: string | null;
|
||||
documentKey: string | null;
|
||||
propertyKey: string | null;
|
||||
}
|
||||
|
||||
export interface ExternalObjectDetection {
|
||||
canonical: ExternalObjectCanonicalUrl;
|
||||
detectorKey: string;
|
||||
providerKey: string;
|
||||
objectType: string;
|
||||
externalId: string;
|
||||
displayKey?: string | null;
|
||||
iconKey?: string | null;
|
||||
displayTitle?: string | null;
|
||||
confidence?: ExternalObjectMentionConfidence;
|
||||
pluginId?: string | null;
|
||||
}
|
||||
|
||||
export interface ExternalObjectDetector {
|
||||
key: string;
|
||||
detect(input: {
|
||||
companyId: string;
|
||||
urls: ExternalObjectCanonicalUrl[];
|
||||
sourceContext: ExternalObjectSourceContext;
|
||||
}): Promise<ExternalObjectDetection[]> | ExternalObjectDetection[];
|
||||
}
|
||||
|
||||
export interface ExternalObjectResolverSnapshot {
|
||||
displayKey?: string | null;
|
||||
iconKey?: string | null;
|
||||
displayTitle?: string | null;
|
||||
statusKey?: string | null;
|
||||
statusLabel?: string | null;
|
||||
statusIconKey?: string | null;
|
||||
statusCategory: ExternalObjectStatusCategory;
|
||||
statusTone: ExternalObjectStatusTone;
|
||||
isTerminal?: boolean;
|
||||
data?: Record<string, unknown>;
|
||||
remoteVersion?: string | null;
|
||||
etag?: string | null;
|
||||
ttlSeconds?: number;
|
||||
}
|
||||
|
||||
export type ExternalObjectResolveResult =
|
||||
| { ok: true; snapshot: ExternalObjectResolverSnapshot }
|
||||
| {
|
||||
ok: false;
|
||||
liveness: Extract<ExternalObjectLivenessState, "auth_required" | "unreachable">;
|
||||
errorCode: string;
|
||||
errorMessage?: string | null;
|
||||
retryAfterSeconds?: number;
|
||||
};
|
||||
|
||||
export interface ExternalObjectResolver {
|
||||
providerKey: string;
|
||||
objectType?: string;
|
||||
resolve(input: {
|
||||
companyId: string;
|
||||
object: ExternalObjectRecord;
|
||||
}): Promise<ExternalObjectResolveResult>;
|
||||
}
|
||||
|
||||
type ExternalObjectRecord = typeof externalObjects.$inferSelect;
|
||||
type ExternalObjectMentionRecord = typeof externalObjectMentions.$inferSelect;
|
||||
|
||||
const DEFAULT_REFRESH_TTL_SECONDS = 300;
|
||||
const DEFAULT_RETRY_AFTER_SECONDS = 300;
|
||||
|
||||
function sourceWhere(input: ExternalObjectSourceContext) {
|
||||
const conditions = [
|
||||
eq(externalObjectMentions.companyId, input.companyId),
|
||||
eq(externalObjectMentions.sourceIssueId, input.sourceIssueId),
|
||||
eq(externalObjectMentions.sourceKind, input.sourceKind),
|
||||
];
|
||||
if (input.sourceRecordId) {
|
||||
conditions.push(eq(externalObjectMentions.sourceRecordId, input.sourceRecordId));
|
||||
} else {
|
||||
conditions.push(isNull(externalObjectMentions.sourceRecordId));
|
||||
}
|
||||
if (input.documentKey) {
|
||||
conditions.push(eq(externalObjectMentions.documentKey, input.documentKey));
|
||||
} else {
|
||||
conditions.push(isNull(externalObjectMentions.documentKey));
|
||||
}
|
||||
if (input.propertyKey) {
|
||||
conditions.push(eq(externalObjectMentions.propertyKey, input.propertyKey));
|
||||
} else {
|
||||
conditions.push(isNull(externalObjectMentions.propertyKey));
|
||||
}
|
||||
return and(...conditions);
|
||||
}
|
||||
|
||||
function addSeconds(date: Date, seconds: number) {
|
||||
return new Date(date.getTime() + Math.max(1, seconds) * 1000);
|
||||
}
|
||||
|
||||
function visibleLiveness(object: ExternalObjectRecord, now = new Date()): ExternalObjectLivenessState {
|
||||
if (object.liveness === "fresh" && object.nextRefreshAt && object.nextRefreshAt <= now) {
|
||||
return "stale";
|
||||
}
|
||||
return object.liveness;
|
||||
}
|
||||
|
||||
function objectChanged(before: ExternalObjectRecord, after: ExternalObjectRecord) {
|
||||
return (
|
||||
before.statusKey !== after.statusKey ||
|
||||
before.statusLabel !== after.statusLabel ||
|
||||
before.statusIconKey !== after.statusIconKey ||
|
||||
before.statusCategory !== after.statusCategory ||
|
||||
before.statusTone !== after.statusTone ||
|
||||
before.isTerminal !== after.isTerminal
|
||||
);
|
||||
}
|
||||
|
||||
function sanitizeErrorMessage(message: string | null | undefined) {
|
||||
if (!message) return null;
|
||||
return message
|
||||
.replace(/https?:\/\/[^\s<>()]+/gi, "[redacted-url]")
|
||||
.replace(/\b(token|key|secret|authorization|bearer)=\S+/gi, "$1=[redacted]");
|
||||
}
|
||||
|
||||
function genericUrlDetector(): ExternalObjectDetector {
|
||||
return {
|
||||
key: "generic-url",
|
||||
detect({ urls }) {
|
||||
return urls.map((canonical) => ({
|
||||
canonical,
|
||||
detectorKey: "generic-url",
|
||||
providerKey: "url",
|
||||
objectType: "link",
|
||||
externalId: canonical.canonicalIdentityHash,
|
||||
displayTitle: canonical.sanitizedDisplayUrl,
|
||||
confidence: "possible",
|
||||
}));
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function createExternalObjectDetectorRegistry(detectors: ExternalObjectDetector[] = []) {
|
||||
const entries = [...detectors, genericUrlDetector()];
|
||||
|
||||
async function detect(input: {
|
||||
companyId: string;
|
||||
urls: ExternalObjectCanonicalUrl[];
|
||||
sourceContext: ExternalObjectSourceContext;
|
||||
}) {
|
||||
const claimed = new Set<string>();
|
||||
const detections: ExternalObjectDetection[] = [];
|
||||
for (const detector of entries) {
|
||||
const remaining = input.urls.filter((url) => !claimed.has(url.canonicalIdentityHash));
|
||||
if (remaining.length === 0) break;
|
||||
try {
|
||||
const detected = await detector.detect({ ...input, urls: remaining });
|
||||
for (const detection of detected) {
|
||||
if (claimed.has(detection.canonical.canonicalIdentityHash)) continue;
|
||||
claimed.add(detection.canonical.canonicalIdentityHash);
|
||||
detections.push({ ...detection, detectorKey: detection.detectorKey || detector.key });
|
||||
}
|
||||
} catch (err) {
|
||||
logger.warn({ err, detectorKey: detector.key }, "external object detector failed");
|
||||
}
|
||||
}
|
||||
return detections;
|
||||
}
|
||||
|
||||
return { detect };
|
||||
}
|
||||
|
||||
export function createExternalObjectResolverRegistry(resolvers: ExternalObjectResolver[] = []) {
|
||||
function find(object: Pick<ExternalObjectRecord, "providerKey" | "objectType">) {
|
||||
return resolvers.find(
|
||||
(resolver) =>
|
||||
resolver.providerKey === object.providerKey &&
|
||||
(!resolver.objectType || resolver.objectType === object.objectType),
|
||||
) ?? null;
|
||||
}
|
||||
return { find };
|
||||
}
|
||||
|
||||
function manifestProvidesObject(
|
||||
manifest: PaperclipPluginManifestV1,
|
||||
object: Pick<ExternalObjectRecord, "providerKey" | "objectType">,
|
||||
) {
|
||||
return (manifest.objectReferences ?? []).some(
|
||||
(provider) =>
|
||||
provider.providerKey === object.providerKey &&
|
||||
provider.objectTypes.includes(object.objectType),
|
||||
);
|
||||
}
|
||||
|
||||
function objectSnapshot(object: ExternalObjectRecord): PluginExternalObjectRecordSnapshot {
|
||||
return {
|
||||
id: object.id,
|
||||
companyId: object.companyId,
|
||||
providerKey: object.providerKey,
|
||||
objectType: object.objectType,
|
||||
externalId: object.externalId,
|
||||
sanitizedCanonicalUrl: object.sanitizedCanonicalUrl,
|
||||
canonicalIdentityHash: object.canonicalIdentityHash,
|
||||
displayKey: object.displayKey,
|
||||
iconKey: object.iconKey,
|
||||
displayTitle: object.displayTitle,
|
||||
statusKey: object.statusKey,
|
||||
statusLabel: object.statusLabel,
|
||||
statusIconKey: object.statusIconKey,
|
||||
statusCategory: object.statusCategory,
|
||||
statusTone: object.statusTone,
|
||||
liveness: object.liveness,
|
||||
isTerminal: object.isTerminal,
|
||||
data: object.data as Record<string, unknown>,
|
||||
remoteVersion: object.remoteVersion,
|
||||
etag: object.etag,
|
||||
};
|
||||
}
|
||||
|
||||
async function readyObjectReferencePlugins(db: Db) {
|
||||
return db
|
||||
.select({
|
||||
id: plugins.id,
|
||||
pluginKey: plugins.pluginKey,
|
||||
manifestJson: plugins.manifestJson,
|
||||
})
|
||||
.from(plugins)
|
||||
.where(eq(plugins.status, "ready"))
|
||||
.orderBy(asc(plugins.installOrder))
|
||||
.then((rows) =>
|
||||
rows.filter((row) => (row.manifestJson.objectReferences?.length ?? 0) > 0),
|
||||
);
|
||||
}
|
||||
|
||||
function createPluginProviderDetector(
|
||||
db: Db,
|
||||
pluginWorkerManager: PluginWorkerManager,
|
||||
): ExternalObjectDetector {
|
||||
return {
|
||||
key: "plugin-object-reference-providers",
|
||||
async detect(input) {
|
||||
const providers = await readyObjectReferencePlugins(db);
|
||||
const detections: ExternalObjectDetection[] = [];
|
||||
|
||||
for (const provider of providers) {
|
||||
const manifest = provider.manifestJson;
|
||||
if (!manifest.capabilities.includes("external.objects.detect")) continue;
|
||||
try {
|
||||
const result = await pluginWorkerManager.call(provider.id, "detectExternalObjects", {
|
||||
companyId: input.companyId,
|
||||
urls: input.urls.map((url) => ({
|
||||
sanitizedCanonicalUrl: url.sanitizedCanonicalUrl,
|
||||
sanitizedDisplayUrl: url.sanitizedDisplayUrl,
|
||||
canonicalIdentityHash: url.canonicalIdentityHash,
|
||||
canonicalIdentity: url.canonicalIdentity as unknown as Record<string, unknown>,
|
||||
redactedMatchedText: url.redactedMatchedText,
|
||||
})),
|
||||
sourceContext: input.sourceContext,
|
||||
});
|
||||
const urlsByHash = new Map(input.urls.map((url) => [url.canonicalIdentityHash, url]));
|
||||
const declaredProviderKeys = new Set((manifest.objectReferences ?? []).map((entry) => entry.providerKey));
|
||||
|
||||
for (const detection of result.detections ?? []) {
|
||||
const canonical = urlsByHash.get(detection.urlIdentityHash);
|
||||
if (!canonical) continue;
|
||||
if (!declaredProviderKeys.has(detection.providerKey)) continue;
|
||||
const declaration = (manifest.objectReferences ?? []).find((entry) => entry.providerKey === detection.providerKey);
|
||||
if (!declaration?.objectTypes.includes(detection.objectType)) continue;
|
||||
detections.push({
|
||||
canonical,
|
||||
detectorKey: `${provider.pluginKey}:${detection.providerKey}`,
|
||||
providerKey: detection.providerKey,
|
||||
objectType: detection.objectType,
|
||||
externalId: detection.externalId,
|
||||
displayKey: detection.displayKey,
|
||||
iconKey: detection.iconKey,
|
||||
displayTitle: detection.displayTitle,
|
||||
confidence: detection.confidence ?? "exact",
|
||||
pluginId: provider.id,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
logger.warn(
|
||||
{ err, pluginId: provider.id, pluginKey: provider.pluginKey },
|
||||
"plugin external object detector failed",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return detections;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function resolveViaPluginProvider(
|
||||
db: Db,
|
||||
pluginWorkerManager: PluginWorkerManager | undefined,
|
||||
object: ExternalObjectRecord,
|
||||
): Promise<PluginExternalObjectResolveResult | null> {
|
||||
if (!pluginWorkerManager) return null;
|
||||
const providers = await readyObjectReferencePlugins(db);
|
||||
for (const provider of providers) {
|
||||
const manifest = provider.manifestJson;
|
||||
if (!manifest.capabilities.includes("external.objects.read")) continue;
|
||||
if (!manifestProvidesObject(manifest, object)) continue;
|
||||
try {
|
||||
return await pluginWorkerManager.call(provider.id, "resolveExternalObject", {
|
||||
companyId: object.companyId,
|
||||
providerKey: object.providerKey,
|
||||
objectType: object.objectType,
|
||||
externalId: object.externalId,
|
||||
object: objectSnapshot(object),
|
||||
});
|
||||
} catch (err) {
|
||||
logger.warn(
|
||||
{ err, pluginId: provider.id, pluginKey: provider.pluginKey, objectId: object.id },
|
||||
"plugin external object resolver failed",
|
||||
);
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "unreachable",
|
||||
errorCode: "plugin_resolver_failed",
|
||||
errorMessage: err instanceof Error ? err.message : String(err),
|
||||
};
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function externalObjectService(
|
||||
db: Db,
|
||||
opts: {
|
||||
detectors?: ExternalObjectDetector[];
|
||||
resolvers?: ExternalObjectResolver[];
|
||||
pluginWorkerManager?: PluginWorkerManager;
|
||||
github?: GitHubExternalObjectProviderOptions | false;
|
||||
enabled?: boolean | (() => boolean | Promise<boolean>);
|
||||
} = {},
|
||||
) {
|
||||
const githubProvider = opts.github === false ? null : createGitHubExternalObjectProvider(db, opts.github);
|
||||
const pluginProviderDetector = opts.pluginWorkerManager
|
||||
? createPluginProviderDetector(db, opts.pluginWorkerManager)
|
||||
: null;
|
||||
const detectorRegistry = createExternalObjectDetectorRegistry([
|
||||
...(pluginProviderDetector ? [pluginProviderDetector] : []),
|
||||
...(opts.detectors ?? []),
|
||||
...(githubProvider ? [githubProvider.detector] : []),
|
||||
]);
|
||||
const resolverRegistry = createExternalObjectResolverRegistry([
|
||||
...(opts.resolvers ?? []),
|
||||
...(githubProvider?.resolvers ?? []),
|
||||
]);
|
||||
|
||||
async function isEnabled() {
|
||||
if (typeof opts.enabled === "function") return await opts.enabled();
|
||||
return opts.enabled ?? true;
|
||||
}
|
||||
|
||||
function emptySummary() {
|
||||
return summarizeObjectPayloads([]);
|
||||
}
|
||||
|
||||
async function issueById(issueId: string, dbOrTx: any = db) {
|
||||
return dbOrTx
|
||||
.select({
|
||||
id: issues.id,
|
||||
companyId: issues.companyId,
|
||||
title: issues.title,
|
||||
description: issues.description,
|
||||
})
|
||||
.from(issues)
|
||||
.where(eq(issues.id, issueId))
|
||||
.then((rows: Array<{ id: string; companyId: string; title: string; description: string | null }>) => rows[0] ?? null);
|
||||
}
|
||||
|
||||
async function upsertObjectFromDetection(
|
||||
companyId: string,
|
||||
detection: ExternalObjectDetection,
|
||||
dbOrTx: any,
|
||||
): Promise<ExternalObjectRecord> {
|
||||
const now = new Date();
|
||||
const canonical = detection.canonical;
|
||||
const values = {
|
||||
companyId,
|
||||
providerKey: detection.providerKey,
|
||||
pluginId: detection.pluginId ?? null,
|
||||
objectType: detection.objectType,
|
||||
externalId: detection.externalId,
|
||||
sanitizedCanonicalUrl: canonical.sanitizedCanonicalUrl,
|
||||
canonicalIdentityHash: canonical.canonicalIdentityHash,
|
||||
displayKey: detection.displayKey ?? null,
|
||||
iconKey: detection.iconKey ?? null,
|
||||
displayTitle: detection.displayTitle ?? canonical.sanitizedDisplayUrl,
|
||||
nextRefreshAt: now,
|
||||
updatedAt: now,
|
||||
};
|
||||
const inserted = await dbOrTx
|
||||
.insert(externalObjects)
|
||||
.values(values)
|
||||
.onConflictDoUpdate({
|
||||
target: [
|
||||
externalObjects.companyId,
|
||||
externalObjects.providerKey,
|
||||
externalObjects.objectType,
|
||||
externalObjects.externalId,
|
||||
],
|
||||
set: {
|
||||
sanitizedCanonicalUrl: values.sanitizedCanonicalUrl,
|
||||
canonicalIdentityHash: values.canonicalIdentityHash,
|
||||
displayKey: values.displayKey,
|
||||
iconKey: values.iconKey,
|
||||
displayTitle: values.displayTitle,
|
||||
nextRefreshAt: sql`coalesce(${externalObjects.nextRefreshAt}, now())`,
|
||||
updatedAt: now,
|
||||
},
|
||||
})
|
||||
.returning();
|
||||
return inserted[0]!;
|
||||
}
|
||||
|
||||
async function replaceSourceMentions(
|
||||
input: ExternalObjectSourceContext & { text: string | null | undefined },
|
||||
dbOrTx: any = db,
|
||||
) {
|
||||
const urls = extractExternalObjectCanonicalUrls(input.text ?? "");
|
||||
await dbOrTx.delete(externalObjectMentions).where(sourceWhere(input));
|
||||
if (urls.length === 0) return;
|
||||
|
||||
const detections = await detectorRegistry.detect({
|
||||
companyId: input.companyId,
|
||||
urls,
|
||||
sourceContext: input,
|
||||
});
|
||||
if (detections.length === 0) return;
|
||||
|
||||
const seen = new Set<string>();
|
||||
const values: Array<typeof externalObjectMentions.$inferInsert> = [];
|
||||
for (const detection of detections) {
|
||||
const canonicalHash = detection.canonical.canonicalIdentityHash;
|
||||
const sourceKey = `${detection.providerKey}:${detection.objectType}:${canonicalHash}`;
|
||||
if (seen.has(sourceKey)) continue;
|
||||
seen.add(sourceKey);
|
||||
const object = await upsertObjectFromDetection(input.companyId, detection, dbOrTx);
|
||||
values.push({
|
||||
companyId: input.companyId,
|
||||
sourceIssueId: input.sourceIssueId,
|
||||
sourceKind: input.sourceKind,
|
||||
sourceRecordId: input.sourceRecordId,
|
||||
documentKey: input.documentKey,
|
||||
propertyKey: input.propertyKey,
|
||||
matchedTextRedacted: detection.canonical.redactedMatchedText,
|
||||
sanitizedDisplayUrl: detection.canonical.sanitizedDisplayUrl,
|
||||
canonicalIdentityHash: canonicalHash,
|
||||
canonicalIdentity: detection.canonical.canonicalIdentity as unknown as Record<string, unknown>,
|
||||
objectId: object.id,
|
||||
providerKey: detection.providerKey,
|
||||
detectorKey: detection.detectorKey,
|
||||
objectType: detection.objectType,
|
||||
confidence: detection.confidence ?? "exact",
|
||||
createdByPluginId: detection.pluginId ?? null,
|
||||
});
|
||||
}
|
||||
if (values.length > 0) {
|
||||
await dbOrTx.insert(externalObjectMentions).values(values);
|
||||
}
|
||||
}
|
||||
|
||||
async function syncIssue(issueId: string, dbOrTx: any = db) {
|
||||
if (!(await isEnabled())) return;
|
||||
const runSync = async (tx: any) => {
|
||||
const issue = await issueById(issueId, tx);
|
||||
if (!issue) throw notFound("Issue not found");
|
||||
await replaceSourceMentions({
|
||||
companyId: issue.companyId,
|
||||
sourceIssueId: issue.id,
|
||||
sourceKind: "title",
|
||||
sourceRecordId: null,
|
||||
documentKey: null,
|
||||
propertyKey: null,
|
||||
text: issue.title,
|
||||
}, tx);
|
||||
await replaceSourceMentions({
|
||||
companyId: issue.companyId,
|
||||
sourceIssueId: issue.id,
|
||||
sourceKind: "description",
|
||||
sourceRecordId: null,
|
||||
documentKey: null,
|
||||
propertyKey: null,
|
||||
text: issue.description,
|
||||
}, tx);
|
||||
};
|
||||
return dbOrTx === db ? db.transaction(runSync) : runSync(dbOrTx);
|
||||
}
|
||||
|
||||
async function syncComment(commentId: string, dbOrTx: any = db) {
|
||||
const runSync = async (tx: any) => {
|
||||
if (!(await isEnabled())) return;
|
||||
const comment = await tx
|
||||
.select({
|
||||
id: issueComments.id,
|
||||
companyId: issueComments.companyId,
|
||||
issueId: issueComments.issueId,
|
||||
body: issueComments.body,
|
||||
})
|
||||
.from(issueComments)
|
||||
.where(eq(issueComments.id, commentId))
|
||||
.then((rows: Array<{ id: string; companyId: string; issueId: string; body: string }>) => rows[0] ?? null);
|
||||
if (!comment) {
|
||||
await tx
|
||||
.delete(externalObjectMentions)
|
||||
.where(and(eq(externalObjectMentions.sourceKind, "comment"), eq(externalObjectMentions.sourceRecordId, commentId)));
|
||||
return;
|
||||
}
|
||||
await replaceSourceMentions({
|
||||
companyId: comment.companyId,
|
||||
sourceIssueId: comment.issueId,
|
||||
sourceKind: "comment",
|
||||
sourceRecordId: comment.id,
|
||||
documentKey: null,
|
||||
propertyKey: null,
|
||||
text: comment.body,
|
||||
}, tx);
|
||||
};
|
||||
return dbOrTx === db ? db.transaction(runSync) : runSync(dbOrTx);
|
||||
}
|
||||
|
||||
async function syncDocument(documentId: string, dbOrTx: any = db) {
|
||||
const runSync = async (tx: any) => {
|
||||
if (!(await isEnabled())) return;
|
||||
const document = await tx
|
||||
.select({
|
||||
documentId: documents.id,
|
||||
companyId: documents.companyId,
|
||||
issueId: issueDocuments.issueId,
|
||||
key: issueDocuments.key,
|
||||
body: documents.latestBody,
|
||||
})
|
||||
.from(issueDocuments)
|
||||
.innerJoin(documents, eq(issueDocuments.documentId, documents.id))
|
||||
.where(eq(documents.id, documentId))
|
||||
.then((rows: Array<{ documentId: string; companyId: string; issueId: string; key: string; body: string }>) => rows[0] ?? null);
|
||||
if (!document) {
|
||||
await tx
|
||||
.delete(externalObjectMentions)
|
||||
.where(and(eq(externalObjectMentions.sourceKind, "document"), eq(externalObjectMentions.sourceRecordId, documentId)));
|
||||
return;
|
||||
}
|
||||
await replaceSourceMentions({
|
||||
companyId: document.companyId,
|
||||
sourceIssueId: document.issueId,
|
||||
sourceKind: "document",
|
||||
sourceRecordId: document.documentId,
|
||||
documentKey: document.key,
|
||||
propertyKey: null,
|
||||
text: document.body,
|
||||
}, tx);
|
||||
};
|
||||
return dbOrTx === db ? db.transaction(runSync) : runSync(dbOrTx);
|
||||
}
|
||||
|
||||
async function safeSync(label: string, fn: () => Promise<void>) {
|
||||
try {
|
||||
await fn();
|
||||
} catch (err) {
|
||||
logger.warn({ err }, `external object ${label} sync failed`);
|
||||
}
|
||||
}
|
||||
|
||||
async function syncIssueSafely(issueId: string) {
|
||||
await safeSync("issue", () => syncIssue(issueId));
|
||||
}
|
||||
|
||||
async function syncCommentSafely(commentId: string, dbOrTx: any = db) {
|
||||
await safeSync("comment", () => syncComment(commentId, dbOrTx));
|
||||
}
|
||||
|
||||
async function syncDocumentSafely(documentId: string) {
|
||||
await safeSync("document", () => syncDocument(documentId));
|
||||
}
|
||||
|
||||
function toObjectPayload(object: ExternalObjectRecord, now = new Date()) {
|
||||
return {
|
||||
...object,
|
||||
liveness: visibleLiveness(object, now),
|
||||
};
|
||||
}
|
||||
|
||||
async function listForIssue(issueId: string) {
|
||||
if (!(await isEnabled())) return [];
|
||||
const issue = await issueById(issueId);
|
||||
if (!issue) throw notFound("Issue not found");
|
||||
const rows = await db
|
||||
.select({
|
||||
mention: externalObjectMentions,
|
||||
object: externalObjects,
|
||||
})
|
||||
.from(externalObjectMentions)
|
||||
.leftJoin(externalObjects, eq(externalObjectMentions.objectId, externalObjects.id))
|
||||
.where(and(
|
||||
eq(externalObjectMentions.companyId, issue.companyId),
|
||||
eq(externalObjectMentions.sourceIssueId, issue.id),
|
||||
))
|
||||
.orderBy(asc(externalObjectMentions.sourceKind), asc(externalObjectMentions.createdAt));
|
||||
const now = new Date();
|
||||
const grouped = new Map<string, {
|
||||
object: ReturnType<typeof toObjectPayload> | null;
|
||||
mentions: ExternalObjectMentionRecord[];
|
||||
mentionCount: number;
|
||||
sourceLabels: string[];
|
||||
}>();
|
||||
for (const row of rows) {
|
||||
const key = row.object?.id ?? `mention:${row.mention.id}`;
|
||||
const existing = grouped.get(key) ?? {
|
||||
object: row.object ? toObjectPayload(row.object, now) : null,
|
||||
mentions: [],
|
||||
mentionCount: 0,
|
||||
sourceLabels: [],
|
||||
};
|
||||
existing.mentions.push(row.mention);
|
||||
existing.mentionCount += 1;
|
||||
const label = formatExternalObjectMentionSourceLabel({
|
||||
sourceKind: row.mention.sourceKind,
|
||||
documentKey: row.mention.documentKey,
|
||||
propertyKey: row.mention.propertyKey,
|
||||
});
|
||||
if (!existing.sourceLabels.includes(label)) existing.sourceLabels.push(label);
|
||||
grouped.set(key, existing);
|
||||
}
|
||||
return [...grouped.values()];
|
||||
}
|
||||
|
||||
function summarizeObjects(objects: Array<ReturnType<typeof toObjectPayload>>) {
|
||||
const byStatusCategory: Record<string, number> = {};
|
||||
const byLiveness: Record<string, number> = {};
|
||||
let highestSeverity: ExternalObjectStatusTone = "neutral";
|
||||
const severityRank: Record<ExternalObjectStatusTone, number> = {
|
||||
neutral: 0,
|
||||
muted: 0,
|
||||
success: 1,
|
||||
info: 2,
|
||||
warning: 3,
|
||||
danger: 4,
|
||||
};
|
||||
for (const object of objects) {
|
||||
byStatusCategory[object.statusCategory] = (byStatusCategory[object.statusCategory] ?? 0) + 1;
|
||||
byLiveness[object.liveness] = (byLiveness[object.liveness] ?? 0) + 1;
|
||||
const livenessTone = object.liveness === "auth_required" || object.liveness === "unreachable"
|
||||
? "danger"
|
||||
: object.liveness === "stale"
|
||||
? "warning"
|
||||
: object.statusTone;
|
||||
if (severityRank[livenessTone] > severityRank[highestSeverity]) highestSeverity = livenessTone;
|
||||
}
|
||||
return {
|
||||
total: objects.length,
|
||||
byStatusCategory,
|
||||
byLiveness,
|
||||
highestSeverity,
|
||||
staleCount: byLiveness.stale ?? 0,
|
||||
authRequiredCount: byLiveness.auth_required ?? 0,
|
||||
unreachableCount: byLiveness.unreachable ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
function summarizeObjectPayloads(objects: Array<ReturnType<typeof toObjectPayload>>, objectLimit = objects.length) {
|
||||
return {
|
||||
...summarizeObjects(objects),
|
||||
objects: objects.slice(0, objectLimit).map((object) => ({
|
||||
id: object.id,
|
||||
providerKey: object.providerKey,
|
||||
objectType: object.objectType,
|
||||
displayKey: object.displayKey,
|
||||
iconKey: object.iconKey,
|
||||
displayTitle: object.displayTitle,
|
||||
statusIconKey: object.statusIconKey,
|
||||
statusCategory: object.statusCategory,
|
||||
statusTone: object.statusTone,
|
||||
liveness: object.liveness,
|
||||
isTerminal: object.isTerminal,
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
async function getIssueSummary(issueId: string) {
|
||||
if (!(await isEnabled())) return emptySummary();
|
||||
const groups = await listForIssue(issueId);
|
||||
const objects = groups.flatMap((group) => (group.object ? [group.object] : []));
|
||||
return summarizeObjectPayloads(objects);
|
||||
}
|
||||
|
||||
async function getIssueSummaries(companyId: string, issueIds: string[]) {
|
||||
if (!(await isEnabled())) return new Map<string, ReturnType<typeof summarizeObjectPayloads>>();
|
||||
const uniqueIssueIds = [...new Set(issueIds)].filter((id) => id.length > 0);
|
||||
const summaries = new Map<string, ReturnType<typeof summarizeObjectPayloads>>();
|
||||
if (uniqueIssueIds.length === 0) return summaries;
|
||||
|
||||
const rows = await db
|
||||
.select({
|
||||
issueId: externalObjectMentions.sourceIssueId,
|
||||
object: externalObjects,
|
||||
})
|
||||
.from(externalObjectMentions)
|
||||
.innerJoin(externalObjects, eq(externalObjectMentions.objectId, externalObjects.id))
|
||||
.where(and(
|
||||
eq(externalObjectMentions.companyId, companyId),
|
||||
inArray(externalObjectMentions.sourceIssueId, uniqueIssueIds),
|
||||
));
|
||||
|
||||
const now = new Date();
|
||||
const objectsByIssueId = new Map<string, Map<string, ReturnType<typeof toObjectPayload>>>();
|
||||
for (const row of rows) {
|
||||
const issueObjects = objectsByIssueId.get(row.issueId) ?? new Map<string, ReturnType<typeof toObjectPayload>>();
|
||||
issueObjects.set(row.object.id, toObjectPayload(row.object, now));
|
||||
objectsByIssueId.set(row.issueId, issueObjects);
|
||||
}
|
||||
|
||||
for (const [issueId, issueObjects] of objectsByIssueId) {
|
||||
const objects = [...issueObjects.values()];
|
||||
if (objects.length > 0) summaries.set(issueId, summarizeObjectPayloads(objects));
|
||||
}
|
||||
return summaries;
|
||||
}
|
||||
|
||||
async function getProjectSummary(projectId: string) {
|
||||
if (!(await isEnabled())) return emptySummary();
|
||||
const projectIssues = await db
|
||||
.select({ id: issues.id, companyId: issues.companyId })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.projectId, projectId), inArray(issues.status, ["todo", "in_progress", "in_review", "blocked"])));
|
||||
if (projectIssues.length === 0) return { ...summarizeObjects([]), objects: [] };
|
||||
const companyIds = new Set(projectIssues.map((issue) => issue.companyId));
|
||||
if (companyIds.size !== 1) return { ...summarizeObjects([]), objects: [] };
|
||||
const issueIds = projectIssues.map((issue) => issue.id);
|
||||
const rows = await db
|
||||
.select({ object: externalObjects })
|
||||
.from(externalObjectMentions)
|
||||
.innerJoin(externalObjects, eq(externalObjectMentions.objectId, externalObjects.id))
|
||||
.where(and(
|
||||
eq(externalObjectMentions.companyId, projectIssues[0]!.companyId),
|
||||
inArray(externalObjectMentions.sourceIssueId, issueIds),
|
||||
));
|
||||
const now = new Date();
|
||||
const objectsById = new Map<string, ReturnType<typeof toObjectPayload>>();
|
||||
for (const row of rows) objectsById.set(row.object.id, toObjectPayload(row.object, now));
|
||||
const objects = [...objectsById.values()];
|
||||
return summarizeObjectPayloads(objects, 25);
|
||||
}
|
||||
|
||||
async function refreshObject(
|
||||
objectId: string,
|
||||
input: {
|
||||
companyId: string;
|
||||
actor?: Pick<LogActivityInput, "actorType" | "actorId" | "agentId" | "runId">;
|
||||
force?: boolean;
|
||||
now?: Date;
|
||||
},
|
||||
) {
|
||||
const now = input.now ?? new Date();
|
||||
const object = await db
|
||||
.select()
|
||||
.from(externalObjects)
|
||||
.where(and(eq(externalObjects.id, objectId), eq(externalObjects.companyId, input.companyId)))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!object) throw notFound("External object not found");
|
||||
if (!input.force && object.nextRefreshAt && object.nextRefreshAt > now) {
|
||||
return { object: toObjectPayload(object, now), refreshed: false, reason: "backoff" as const };
|
||||
}
|
||||
|
||||
const pluginResult = await resolveViaPluginProvider(db, opts.pluginWorkerManager, object);
|
||||
const resolver = pluginResult ? null : resolverRegistry.find(object);
|
||||
if (!pluginResult && !resolver) {
|
||||
const [updated] = await db
|
||||
.update(externalObjects)
|
||||
.set({
|
||||
liveness: visibleLiveness(object, now) === "fresh" ? "stale" : object.liveness,
|
||||
nextRefreshAt: addSeconds(now, DEFAULT_RETRY_AFTER_SECONDS),
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(and(eq(externalObjects.id, object.id), eq(externalObjects.companyId, object.companyId)))
|
||||
.returning();
|
||||
return { object: toObjectPayload(updated ?? object, now), refreshed: false, reason: "no_resolver" as const };
|
||||
}
|
||||
|
||||
const result = pluginResult ?? await resolver!.resolve({ companyId: object.companyId, object });
|
||||
if (!result.ok) {
|
||||
const [updated] = await db
|
||||
.update(externalObjects)
|
||||
.set({
|
||||
liveness: result.liveness,
|
||||
lastErrorAt: now,
|
||||
lastErrorCode: result.errorCode,
|
||||
lastErrorMessage: sanitizeErrorMessage(result.errorMessage),
|
||||
nextRefreshAt: addSeconds(now, result.retryAfterSeconds ?? DEFAULT_RETRY_AFTER_SECONDS),
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(and(eq(externalObjects.id, object.id), eq(externalObjects.companyId, object.companyId)))
|
||||
.returning();
|
||||
publishLiveEvent({
|
||||
companyId: object.companyId,
|
||||
type: "external_object.updated",
|
||||
payload: { objectId: object.id, liveness: result.liveness },
|
||||
});
|
||||
return { object: toObjectPayload(updated ?? object, now), refreshed: true, reason: result.liveness };
|
||||
}
|
||||
|
||||
const snapshot = result.snapshot;
|
||||
const patch = {
|
||||
displayKey: snapshot.displayKey ?? object.displayKey,
|
||||
iconKey: snapshot.iconKey ?? object.iconKey,
|
||||
displayTitle: snapshot.displayTitle ?? object.displayTitle,
|
||||
statusKey: snapshot.statusKey ?? object.statusKey,
|
||||
statusLabel: snapshot.statusLabel ?? object.statusLabel,
|
||||
statusIconKey: snapshot.statusIconKey ?? object.statusIconKey,
|
||||
statusCategory: snapshot.statusCategory,
|
||||
statusTone: snapshot.statusTone,
|
||||
isTerminal: snapshot.isTerminal ?? object.isTerminal,
|
||||
data: snapshot.data ?? object.data,
|
||||
remoteVersion: snapshot.remoteVersion ?? object.remoteVersion,
|
||||
etag: snapshot.etag ?? object.etag,
|
||||
liveness: "fresh" as ExternalObjectLivenessState,
|
||||
lastResolvedAt: now,
|
||||
lastErrorAt: null,
|
||||
lastErrorCode: null,
|
||||
lastErrorMessage: null,
|
||||
nextRefreshAt: addSeconds(now, snapshot.ttlSeconds ?? DEFAULT_REFRESH_TTL_SECONDS),
|
||||
updatedAt: now,
|
||||
};
|
||||
const [updated] = await db
|
||||
.update(externalObjects)
|
||||
.set({
|
||||
...patch,
|
||||
lastChangedAt: objectChanged(object, { ...object, ...patch }) ? now : object.lastChangedAt,
|
||||
})
|
||||
.where(and(eq(externalObjects.id, object.id), eq(externalObjects.companyId, object.companyId)))
|
||||
.returning();
|
||||
const next = updated ?? object;
|
||||
if (objectChanged(object, next) && input.actor) {
|
||||
await logActivity(db, {
|
||||
companyId: object.companyId,
|
||||
actorType: input.actor.actorType,
|
||||
actorId: input.actor.actorId,
|
||||
agentId: input.actor.agentId,
|
||||
runId: input.actor.runId,
|
||||
action: "external_object.status_changed",
|
||||
entityType: "external_object",
|
||||
entityId: object.id,
|
||||
details: {
|
||||
providerKey: object.providerKey,
|
||||
objectType: object.objectType,
|
||||
statusCategory: next.statusCategory,
|
||||
statusLabel: next.statusLabel,
|
||||
_previous: {
|
||||
statusCategory: object.statusCategory,
|
||||
statusLabel: object.statusLabel,
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
publishLiveEvent({
|
||||
companyId: object.companyId,
|
||||
type: "external_object.updated",
|
||||
payload: { objectId: object.id, statusCategory: next.statusCategory, liveness: next.liveness },
|
||||
});
|
||||
return { object: toObjectPayload(next, now), refreshed: true, reason: "resolved" as const };
|
||||
}
|
||||
|
||||
async function refreshIssueObjects(issueId: string, input: {
|
||||
companyId: string;
|
||||
objectIds?: string[];
|
||||
actor?: Pick<LogActivityInput, "actorType" | "actorId" | "agentId" | "runId">;
|
||||
}) {
|
||||
if (!(await isEnabled())) return [];
|
||||
const groups = await listForIssue(issueId);
|
||||
const objectIds = groups
|
||||
.flatMap((group) => (group.object ? [group.object.id] : []))
|
||||
.filter((id) => !input.objectIds || input.objectIds.includes(id));
|
||||
const results = [];
|
||||
for (const objectId of objectIds) {
|
||||
results.push(await refreshObject(objectId, { companyId: input.companyId, actor: input.actor }));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
async function refreshDueObjects(companyId: string, limit = 50, now = new Date()) {
|
||||
if (!(await isEnabled())) return [];
|
||||
const due = await db
|
||||
.select({ id: externalObjects.id })
|
||||
.from(externalObjects)
|
||||
.where(
|
||||
and(
|
||||
eq(externalObjects.companyId, companyId),
|
||||
eq(externalObjects.isTerminal, false),
|
||||
lte(externalObjects.nextRefreshAt, now),
|
||||
),
|
||||
)
|
||||
.limit(limit);
|
||||
const results = [];
|
||||
for (const row of due) {
|
||||
results.push(await refreshObject(row.id, {
|
||||
companyId,
|
||||
actor: { actorType: "system", actorId: "external-object-resolver", agentId: null, runId: null },
|
||||
now,
|
||||
}));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
return {
|
||||
syncIssue,
|
||||
syncComment,
|
||||
syncDocument,
|
||||
syncIssueSafely,
|
||||
syncCommentSafely,
|
||||
syncDocumentSafely,
|
||||
listForIssue,
|
||||
getIssueSummary,
|
||||
getIssueSummaries,
|
||||
getProjectSummary,
|
||||
refreshObject,
|
||||
refreshIssueObjects,
|
||||
refreshDueObjects,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,445 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import type { ExternalObjectCanonicalUrl } from "@paperclipai/shared";
|
||||
import { ghFetch, gitHubApiBase } from "./github-fetch.js";
|
||||
import { secretService } from "./secrets.js";
|
||||
import type {
|
||||
ExternalObjectDetection,
|
||||
ExternalObjectDetector,
|
||||
ExternalObjectResolver,
|
||||
ExternalObjectResolverSnapshot,
|
||||
ExternalObjectResolveResult,
|
||||
} from "./external-objects.js";
|
||||
|
||||
type FetchLike = (url: string, init?: RequestInit) => Promise<Response>;
|
||||
|
||||
export interface GitHubExternalObjectProviderOptions {
|
||||
fetch?: FetchLike;
|
||||
tokenProvider?: (companyId: string) => Promise<string | null> | string | null;
|
||||
secretNames?: readonly string[];
|
||||
}
|
||||
|
||||
interface GitHubObjectIdentity {
|
||||
host: string;
|
||||
owner: string;
|
||||
repo: string;
|
||||
number: number;
|
||||
objectType: "pull_request" | "issue";
|
||||
pathKind: "pull" | "issues";
|
||||
}
|
||||
|
||||
const DEFAULT_GITHUB_TOKEN_SECRET_NAMES = ["GITHUB_TOKEN", "GH_TOKEN", "PAPERCLIP_GITHUB_TOKEN"] as const;
|
||||
const GITHUB_OBJECT_TTL_SECONDS = 300;
|
||||
|
||||
function isGitHubHost(host: string) {
|
||||
const h = host.toLowerCase();
|
||||
return h === "github.com" || h === "www.github.com";
|
||||
}
|
||||
|
||||
function asRecord(value: unknown): Record<string, unknown> | null {
|
||||
return value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : null;
|
||||
}
|
||||
|
||||
function asString(value: unknown) {
|
||||
return typeof value === "string" ? value : null;
|
||||
}
|
||||
|
||||
function asBoolean(value: unknown) {
|
||||
return typeof value === "boolean" ? value : null;
|
||||
}
|
||||
|
||||
function asNestedString(record: Record<string, unknown>, key: string, nestedKey: string) {
|
||||
const nested = asRecord(record[key]);
|
||||
return nested ? asString(nested[nestedKey]) : null;
|
||||
}
|
||||
|
||||
function parseGitHubCanonicalUrl(canonical: ExternalObjectCanonicalUrl): GitHubObjectIdentity | null {
|
||||
if (canonical.canonicalIdentity.scheme !== "https") return null;
|
||||
const host = canonical.canonicalIdentity.host.toLowerCase();
|
||||
if (!isGitHubHost(host)) return null;
|
||||
|
||||
const parts = canonical.canonicalIdentity.path.split("/").filter(Boolean);
|
||||
if (parts.length !== 4) return null;
|
||||
const [owner, repo, kind, rawNumber] = parts;
|
||||
if (!owner || !repo || !kind || !rawNumber) return null;
|
||||
if (!/^[A-Za-z0-9_.-]+$/.test(owner) || !/^[A-Za-z0-9_.-]+$/.test(repo)) return null;
|
||||
if (!/^[1-9][0-9]*$/.test(rawNumber)) return null;
|
||||
if (kind !== "pull" && kind !== "issues") return null;
|
||||
|
||||
return {
|
||||
host: host === "www.github.com" ? "github.com" : host,
|
||||
owner,
|
||||
repo,
|
||||
number: Number(rawNumber),
|
||||
pathKind: kind,
|
||||
objectType: kind === "pull" ? "pull_request" : "issue",
|
||||
};
|
||||
}
|
||||
|
||||
function parseGitHubObject(object: { externalId: string; sanitizedCanonicalUrl: string | null }): GitHubObjectIdentity | null {
|
||||
const match = /^([^/]+)\/([^/]+)#(pull|issues)\/([1-9][0-9]*)$/.exec(object.externalId);
|
||||
if (!match) return null;
|
||||
let host = "github.com";
|
||||
if (object.sanitizedCanonicalUrl) {
|
||||
try {
|
||||
const url = new URL(object.sanitizedCanonicalUrl);
|
||||
if (isGitHubHost(url.hostname)) host = url.hostname === "www.github.com" ? "github.com" : url.hostname;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
return {
|
||||
host,
|
||||
owner: match[1]!,
|
||||
repo: match[2]!,
|
||||
pathKind: match[3] as "pull" | "issues",
|
||||
number: Number(match[4]),
|
||||
objectType: match[3] === "pull" ? "pull_request" : "issue",
|
||||
};
|
||||
}
|
||||
|
||||
function externalIdFor(identity: GitHubObjectIdentity) {
|
||||
return `${identity.owner.toLowerCase()}/${identity.repo.toLowerCase()}#${identity.pathKind}/${identity.number}`;
|
||||
}
|
||||
|
||||
function displayTitleFor(identity: GitHubObjectIdentity) {
|
||||
return `${identity.owner}/${identity.repo}#${identity.number}`;
|
||||
}
|
||||
|
||||
function displayKeyFor(identity: Pick<GitHubObjectIdentity, "objectType">) {
|
||||
return identity.objectType === "pull_request" ? "GitHub Pull Request" : "GitHub Issue";
|
||||
}
|
||||
|
||||
function retryAfterSeconds(response: Response) {
|
||||
const retryAfter = response.headers.get("retry-after");
|
||||
if (retryAfter && /^[0-9]+$/.test(retryAfter)) return Number(retryAfter);
|
||||
|
||||
const reset = response.headers.get("x-ratelimit-reset");
|
||||
if (reset && /^[0-9]+$/.test(reset)) {
|
||||
return Math.max(1, Number(reset) - Math.floor(Date.now() / 1000));
|
||||
}
|
||||
|
||||
return 300;
|
||||
}
|
||||
|
||||
function failureFromGitHubResponse(response: Response): ExternalObjectResolveResult | null {
|
||||
if (response.status === 401) {
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "auth_required",
|
||||
errorCode: "github_auth_required",
|
||||
errorMessage: "GitHub authentication is required to refresh this object.",
|
||||
retryAfterSeconds: retryAfterSeconds(response),
|
||||
};
|
||||
}
|
||||
|
||||
if (response.status === 403) {
|
||||
const rateLimitRemaining = response.headers.get("x-ratelimit-remaining");
|
||||
if (rateLimitRemaining === "0") {
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "unreachable",
|
||||
errorCode: "github_rate_limited",
|
||||
errorMessage: "GitHub rate limit reached while refreshing this object.",
|
||||
retryAfterSeconds: retryAfterSeconds(response),
|
||||
};
|
||||
}
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "auth_required",
|
||||
errorCode: "github_forbidden",
|
||||
errorMessage: "GitHub rejected the configured credentials for this object.",
|
||||
retryAfterSeconds: retryAfterSeconds(response),
|
||||
};
|
||||
}
|
||||
|
||||
if (response.status === 429 || response.status >= 500) {
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "unreachable",
|
||||
errorCode: response.status === 429 ? "github_rate_limited" : "github_unreachable",
|
||||
errorMessage: `GitHub returned HTTP ${response.status} while refreshing this object.`,
|
||||
retryAfterSeconds: retryAfterSeconds(response),
|
||||
};
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function notFoundSnapshot(identity: GitHubObjectIdentity, etag: string | null): ExternalObjectResolverSnapshot {
|
||||
return {
|
||||
displayKey: displayKeyFor(identity),
|
||||
iconKey: "github",
|
||||
displayTitle: displayTitleFor(identity),
|
||||
statusKey: "not_found",
|
||||
statusLabel: "Not found",
|
||||
statusIconKey: "archive",
|
||||
statusCategory: "archived",
|
||||
statusTone: "muted",
|
||||
isTerminal: true,
|
||||
etag,
|
||||
ttlSeconds: GITHUB_OBJECT_TTL_SECONDS,
|
||||
data: {
|
||||
provider: "github",
|
||||
owner: identity.owner,
|
||||
repo: identity.repo,
|
||||
number: identity.number,
|
||||
notFound: true,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function pullRequestSnapshot(identity: GitHubObjectIdentity, body: Record<string, unknown>, etag: string | null): ExternalObjectResolverSnapshot {
|
||||
const title = asString(body.title);
|
||||
const state = asString(body.state) ?? "unknown";
|
||||
const draft = asBoolean(body.draft) ?? false;
|
||||
const merged = (asBoolean(body.merged) ?? false) || Boolean(asString(body.merged_at));
|
||||
const authorLogin = asNestedString(body, "user", "login");
|
||||
const headRef = asNestedString(body, "head", "ref");
|
||||
const baseRef = asNestedString(body, "base", "ref");
|
||||
const reviewDecision = asString(body.review_decision);
|
||||
|
||||
let statusKey = state;
|
||||
let statusLabel = state === "open" ? "Open" : state === "closed" ? "Closed" : "Unknown";
|
||||
let statusCategory: ExternalObjectResolverSnapshot["statusCategory"] = state === "open" ? "open" : "unknown";
|
||||
let statusTone: ExternalObjectResolverSnapshot["statusTone"] = state === "open" ? "info" : "neutral";
|
||||
let isTerminal = false;
|
||||
|
||||
if (merged) {
|
||||
statusKey = "merged";
|
||||
statusLabel = "Merged";
|
||||
statusCategory = "succeeded";
|
||||
statusTone = "success";
|
||||
isTerminal = true;
|
||||
} else if (state === "closed") {
|
||||
statusKey = "closed";
|
||||
statusLabel = "Closed";
|
||||
statusCategory = "closed";
|
||||
statusTone = "muted";
|
||||
isTerminal = true;
|
||||
} else if (draft) {
|
||||
statusKey = "draft";
|
||||
statusLabel = "Draft";
|
||||
statusCategory = "waiting";
|
||||
statusTone = "warning";
|
||||
}
|
||||
|
||||
return {
|
||||
displayKey: displayKeyFor(identity),
|
||||
iconKey: "github",
|
||||
displayTitle: title ? `${displayTitleFor(identity)}: ${title}` : displayTitleFor(identity),
|
||||
statusKey,
|
||||
statusLabel,
|
||||
statusIconKey: merged
|
||||
? "git-merge"
|
||||
: state === "closed"
|
||||
? "x-circle"
|
||||
: draft
|
||||
? "clock"
|
||||
: "git-pull-request",
|
||||
statusCategory,
|
||||
statusTone,
|
||||
isTerminal,
|
||||
remoteVersion: asString(body.updated_at),
|
||||
etag,
|
||||
ttlSeconds: GITHUB_OBJECT_TTL_SECONDS,
|
||||
data: {
|
||||
provider: "github",
|
||||
owner: identity.owner,
|
||||
repo: identity.repo,
|
||||
number: identity.number,
|
||||
state,
|
||||
merged,
|
||||
draft,
|
||||
...(authorLogin ? { authorLogin } : {}),
|
||||
...(headRef ? { headRef } : {}),
|
||||
...(baseRef ? { baseRef } : {}),
|
||||
...(reviewDecision ? { reviewDecision } : {}),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function issueSnapshot(identity: GitHubObjectIdentity, body: Record<string, unknown>, etag: string | null): ExternalObjectResolverSnapshot {
|
||||
const title = asString(body.title);
|
||||
const state = asString(body.state) ?? "unknown";
|
||||
const stateReason = asString(body.state_reason);
|
||||
const authorLogin = asNestedString(body, "user", "login");
|
||||
const statusKey = state === "closed" && stateReason ? `closed_${stateReason}` : state;
|
||||
const statusLabel = state === "closed"
|
||||
? stateReason
|
||||
? `Closed: ${stateReason.replace(/_/g, " ")}`
|
||||
: "Closed"
|
||||
: state === "open"
|
||||
? "Open"
|
||||
: "Unknown";
|
||||
|
||||
return {
|
||||
displayKey: displayKeyFor(identity),
|
||||
iconKey: "github",
|
||||
displayTitle: title ? `${displayTitleFor(identity)}: ${title}` : displayTitleFor(identity),
|
||||
statusKey,
|
||||
statusLabel,
|
||||
statusIconKey: state === "closed" ? "circle" : "circle-dot",
|
||||
statusCategory: state === "open" ? "open" : state === "closed" ? "closed" : "unknown",
|
||||
statusTone: state === "open" ? "info" : state === "closed" ? "muted" : "neutral",
|
||||
isTerminal: state === "closed",
|
||||
remoteVersion: asString(body.updated_at),
|
||||
etag,
|
||||
ttlSeconds: GITHUB_OBJECT_TTL_SECONDS,
|
||||
data: {
|
||||
provider: "github",
|
||||
owner: identity.owner,
|
||||
repo: identity.repo,
|
||||
number: identity.number,
|
||||
state,
|
||||
...(stateReason ? { stateReason } : {}),
|
||||
...(authorLogin ? { authorLogin } : {}),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function safeJson(response: Response) {
|
||||
try {
|
||||
return asRecord(await response.json());
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function defaultTokenProvider(db: Db, companyId: string, secretNames: readonly string[]) {
|
||||
const secrets = secretService(db);
|
||||
for (const secretName of secretNames) {
|
||||
const secret = await secrets.getByName(companyId, secretName);
|
||||
if (!secret) continue;
|
||||
const token = await secrets.resolveSecretValue(companyId, secret.id, "latest");
|
||||
const trimmed = token.trim();
|
||||
if (trimmed) return trimmed;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function createGitHubExternalObjectProvider(
|
||||
db: Db,
|
||||
opts: GitHubExternalObjectProviderOptions = {},
|
||||
): { detector: ExternalObjectDetector; resolvers: ExternalObjectResolver[] } {
|
||||
const fetchImpl = opts.fetch ?? ghFetch;
|
||||
const secretNames = opts.secretNames ?? DEFAULT_GITHUB_TOKEN_SECRET_NAMES;
|
||||
const tokenProvider = Object.prototype.hasOwnProperty.call(opts, "tokenProvider") && opts.tokenProvider !== undefined
|
||||
? opts.tokenProvider
|
||||
: ((companyId: string) => defaultTokenProvider(db, companyId, secretNames));
|
||||
|
||||
const detector: ExternalObjectDetector = {
|
||||
key: "github",
|
||||
detect({ urls }): ExternalObjectDetection[] {
|
||||
return urls.flatMap((canonical) => {
|
||||
const identity = parseGitHubCanonicalUrl(canonical);
|
||||
if (!identity) return [];
|
||||
return [{
|
||||
canonical,
|
||||
detectorKey: "github",
|
||||
providerKey: "github",
|
||||
objectType: identity.objectType,
|
||||
externalId: externalIdFor(identity),
|
||||
displayKey: displayKeyFor(identity),
|
||||
iconKey: "github",
|
||||
displayTitle: displayTitleFor(identity),
|
||||
confidence: "exact",
|
||||
}];
|
||||
});
|
||||
},
|
||||
};
|
||||
|
||||
function resolver(objectType: GitHubObjectIdentity["objectType"]): ExternalObjectResolver {
|
||||
return {
|
||||
providerKey: "github",
|
||||
objectType,
|
||||
async resolve({ companyId, object }) {
|
||||
const identity = parseGitHubObject(object);
|
||||
if (!identity || identity.objectType !== objectType) {
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "unreachable",
|
||||
errorCode: "github_invalid_identity",
|
||||
errorMessage: "GitHub object identity is invalid.",
|
||||
retryAfterSeconds: GITHUB_OBJECT_TTL_SECONDS,
|
||||
};
|
||||
}
|
||||
|
||||
let token: string | null = null;
|
||||
try {
|
||||
token = typeof tokenProvider === "function" ? await tokenProvider(companyId) : tokenProvider;
|
||||
} catch {
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "auth_required",
|
||||
errorCode: "github_token_unavailable",
|
||||
errorMessage: "Configured GitHub credentials could not be resolved.",
|
||||
retryAfterSeconds: GITHUB_OBJECT_TTL_SECONDS,
|
||||
};
|
||||
}
|
||||
token = token?.trim() || null;
|
||||
const headers: Record<string, string> = {
|
||||
accept: "application/vnd.github+json",
|
||||
"user-agent": "paperclip-external-object-resolver",
|
||||
"x-github-api-version": "2022-11-28",
|
||||
};
|
||||
if (token) headers.authorization = `Bearer ${token}`;
|
||||
|
||||
const apiKind = objectType === "pull_request" ? "pulls" : "issues";
|
||||
const url = `${gitHubApiBase(identity.host)}/repos/${encodeURIComponent(identity.owner)}/${encodeURIComponent(identity.repo)}/${apiKind}/${identity.number}`;
|
||||
|
||||
let response: Response;
|
||||
try {
|
||||
response = await fetchImpl(url, { headers });
|
||||
} catch {
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "unreachable",
|
||||
errorCode: "github_fetch_failed",
|
||||
errorMessage: "GitHub could not be reached while refreshing this object.",
|
||||
retryAfterSeconds: GITHUB_OBJECT_TTL_SECONDS,
|
||||
};
|
||||
}
|
||||
|
||||
const etag = response.headers.get("etag");
|
||||
if (response.status === 404) {
|
||||
return { ok: true, snapshot: notFoundSnapshot(identity, etag) };
|
||||
}
|
||||
|
||||
const failure = failureFromGitHubResponse(response);
|
||||
if (failure) return failure;
|
||||
if (!response.ok) {
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "unreachable",
|
||||
errorCode: "github_unexpected_response",
|
||||
errorMessage: `GitHub returned HTTP ${response.status} while refreshing this object.`,
|
||||
retryAfterSeconds: GITHUB_OBJECT_TTL_SECONDS,
|
||||
};
|
||||
}
|
||||
|
||||
const body = await safeJson(response);
|
||||
if (!body) {
|
||||
return {
|
||||
ok: false,
|
||||
liveness: "unreachable",
|
||||
errorCode: "github_invalid_response",
|
||||
errorMessage: "GitHub returned an invalid object response.",
|
||||
retryAfterSeconds: GITHUB_OBJECT_TTL_SECONDS,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
snapshot: objectType === "pull_request"
|
||||
? pullRequestSnapshot(identity, body, etag)
|
||||
: issueSnapshot(identity, body, etag),
|
||||
};
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
detector,
|
||||
resolvers: [resolver("pull_request"), resolver("issue")],
|
||||
};
|
||||
}
|
||||
@@ -33,6 +33,15 @@ export {
|
||||
resolveTaskWatchdogMutationScope,
|
||||
taskWatchdogScopeAllowsIssueMutation,
|
||||
} from "./task-watchdog-scope.js";
|
||||
export {
|
||||
createExternalObjectDetectorRegistry,
|
||||
createExternalObjectResolverRegistry,
|
||||
externalObjectService,
|
||||
type ExternalObjectDetector,
|
||||
type ExternalObjectResolver,
|
||||
type ExternalObjectResolveResult,
|
||||
type ExternalObjectResolverSnapshot,
|
||||
} from "./external-objects.js";
|
||||
export { goalService } from "./goals.js";
|
||||
export { activityService, type ActivityFilters } from "./activity.js";
|
||||
export { approvalService } from "./approvals.js";
|
||||
|
||||
@@ -52,6 +52,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta
|
||||
enableExperimentalFileViewer: parsed.data.enableExperimentalFileViewer ?? false,
|
||||
enableTaskWatchdogs: parsed.data.enableTaskWatchdogs ?? false,
|
||||
enableCloudSync: parsed.data.enableCloudSync ?? false,
|
||||
enableExternalObjects: parsed.data.enableExternalObjects ?? false,
|
||||
autoRestartDevServerWhenIdle: parsed.data.autoRestartDevServerWhenIdle ?? false,
|
||||
enableIssueGraphLivenessAutoRecovery: parsed.data.enableIssueGraphLivenessAutoRecovery ?? false,
|
||||
issueGraphLivenessAutoRecoveryLookbackHours:
|
||||
@@ -68,6 +69,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta
|
||||
enableIssuePlanDecompositions: false,
|
||||
enableExperimentalFileViewer: false,
|
||||
enableCloudSync: false,
|
||||
enableExternalObjects: false,
|
||||
autoRestartDevServerWhenIdle: false,
|
||||
enableIssueGraphLivenessAutoRecovery: false,
|
||||
issueGraphLivenessAutoRecoveryLookbackHours:
|
||||
|
||||
@@ -99,6 +99,10 @@ const OPERATION_CAPABILITIES: Record<string, readonly PluginCapability[]> = {
|
||||
"telemetry.track": ["telemetry.track"],
|
||||
"db.migrate": ["database.namespace.migrate"],
|
||||
"db.execute": ["database.namespace.write"],
|
||||
"external.objects.detect": ["external.objects.detect"],
|
||||
"external.objects.read": ["external.objects.read"],
|
||||
"external.objects.write": ["external.objects.write"],
|
||||
"external.objects.refresh": ["external.objects.refresh"],
|
||||
|
||||
// Plugin state operations
|
||||
"plugin.state.get": ["plugin.state.read"],
|
||||
@@ -188,6 +192,7 @@ const FEATURE_CAPABILITIES: Record<string, PluginCapability> = {
|
||||
agents: "agents.managed",
|
||||
projects: "projects.managed",
|
||||
routines: "routines.managed",
|
||||
objectReferences: "external.objects.detect",
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -452,6 +457,14 @@ export function pluginCapabilityValidator(): PluginCapabilityValidator {
|
||||
}
|
||||
}
|
||||
|
||||
if ((manifest.objectReferences?.length ?? 0) > 0) {
|
||||
for (const requiredCap of ["external.objects.detect", "external.objects.read"] as const) {
|
||||
if (!declared.has(requiredCap) && !allMissing.includes(requiredCap)) {
|
||||
allMissing.push(requiredCap);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Check UI slots → required capabilities
|
||||
const uiSlots = manifest.ui?.slots ?? [];
|
||||
if (uiSlots.length > 0) {
|
||||
|
||||
Reference in new issue
Block a user