feat(links): cache rich link status in the database and load it on idle (#21746)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task text contains links to pull requests and other work.
> - A link has useful status that should be visible when a task opens.
> - The current cache lives in the server process and the browser.
> - A new browser or server restart has no stored status to show.
> - This pull request stores link status in the database and reads it in
batches.
> - Idle page work updates stale status without a hover or a delay to
task data.

## Linked Issues or Issue Description

**Problem or motivation**

PR status is unknown in a new browser until the user points to each
link. The process cache cannot supply saved status after a server
restart. Related work: #20593 and #20654. A search found no other PR for
database link caching.

**Proposed solution**

Store status by company, viewer and canonical resource URL. Read saved
status in one indexed batch query. After document load and core request
settlement, use browser idle time to update missing or stale status. Use
the same query for hover and idle work.

**Alternatives considered**

Adding provider calls to task reads would delay task loading.
Company-wide reuse of personal GitHub results would cross viewer access
boundaries. A periodic server worker would create work when no page
needs the result.

**Roadmap alignment**

This extends the shipped rich-link feature. It does not duplicate a
roadmap project or add a new provider integration.

## What Changed

- Add an additive, repeat-safe `link_previews` migration. Store channel,
resource kind, free-form status, JSON metadata and freshness. The table
has no task relationship or GitHub status enum.
- Bound each company/viewer cache to 1,000 rows across concurrent server
instances. Remove rows expired for over 30 days during writes. Limit
provider lookups to 600 requests per minute per scope/process. Keep
cached batch reads available during backoff.
- Add a database store owned by the link service. Reuse canonical
results across tasks. Keep company and viewer isolation. Save hover and
idle results. Preserve last known status during an outage and cache
unavailable results.
- Add a board-only batch GET endpoint, including viewer-role access. It
reads at most 50 URLs with one indexed select and makes no provider
calls. Both endpoints use private, no-store HTTP responses.
- Add a page coordinator for mounted rich links. Reuse browser data,
share pending reads, and defer provider work until document load, core
request settlement and browser idle time. Limit automatic work to 200
unique links and three requests in flight. Remove queued work when links
leave the page. Keep encoded batch URLs below 6 KB.
- Keep exact navigation URLs, status styling, anchored cards and
keyboard access. Add a page-load Storybook case and browser regression.

The link service owns cache policy and provider calls. The browser
coordinator owns page timing. The shared contract adds only the batch
limit and response shape. The database adds storage because process and
browser caches cannot serve a new browser session. Plugin resolver
registration is outside this change; storage accepts future provider
channels and status values.

## Verification

- **All 93 affected tests pass:** 40 link service, real PostgreSQL and
HTTP cases, plus 53 OpenAPI cases. Two SQL regressions and one HTTP case
failed before the storage repair. Concurrent writers now retain at most
1,000 rows per company/viewer; cleanup removes long-expired rows and
keeps recent stale status. Request backoff preserves cached reads and
isolates companies/viewers. The earlier eight concurrent hydration
regressions also pass. Focused store/service/unit types pass. Commands:
`pnpm exec vitest run server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1` and
`pnpm exec vitest run server/src/__tests__/openapi-routes.test.ts
--maxWorkers=1`.
- **All 50 affected server tests passed before the later server-only
repairs** on the master-integrated source: 20 recovery/deadline cases,
one authenticated review-counts HTTP case, 10 link-service cases, 13
real PostgreSQL cache cases and six link-route cases. Command: `pnpm
exec vitest run server/src/__tests__/issue-recovery-deadlines.test.ts
server/src/__tests__/tool-connection-review-counts-http.test.ts
server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1`.
- Reproduced the CI watchdog assertion locally twice before the test
repair. Ordinary assignment recovery can win `wakeReason` while the
watchdog wake coalesces into the same successful run with its full
mandate. The test now verifies the persisted watchdog identity, watched
issue and stopped fingerprint, plus exactly one review run. All 21
recovery/HTTP cases passed before integrating master; all 50 affected
cases pass after integration. No runtime behavior was changed for this
repair.
- **Fresh static Storybook build and all 22 browser cases pass**, with
two workers and retries disabled. They cover states, anchored hover,
keyboard access, exact navigation, duplicate requests, reload
persistence, expiry and saved-status hydration followed by one idle
refresh. The first browser attempt could not launch because this session
lacked Chromium; installing the expected browser resolved setup. That
attempt had no behavioral verdict.
- The cache repair remains covered by four PostgreSQL red/green
regressions for expired memory and newer durable status during provider
failure/timeout, plus check-time precedence and failed-read recovery.
Earlier UI/database/focused server type and migration
safety/compatibility/replay checks remain unchanged-input evidence.
- Master `77ddf260bf9a3de3dcb8456a60f4f809d67401fd` is integrated
cleanly. Link migration 0489 and its snapshot chain are unchanged. SDK
source validation checks 545 inputs with no stale source identities or
changed runtime/lock pins. Whitespace validation passes.
- `pnpm verify --explain` returns exit 2 and requires broad CI. No full
local suite or deployed acceptance is claimed. Earlier local full-server
type checks exceeded memory; current-head CI supplies broad
type/build/test coverage.
- [Previous CI
43900336895](https://github.com/paperclipai/paperclip/actions/runs/43900336895)
passed full types/build but failed on embedded PostgreSQL startup
(`ECONNREFUSED`), the watchdog assertion above, and unrelated plan-task
browser cleanup (activity-log foreign key on company deletion). One
failed-jobs retry was requested. Its browser shard 7 now passes without
code changes; the exact deletion-race cause remains unproven. The
watchdog assertion is repaired in this revision. The previous failures
remain recorded, not counted as an overall pass.
- [Prior CI
43905482848](https://github.com/paperclipai/paperclip/actions/runs/43905482848)
passed full types, build and all eight browser shards. Several test jobs
were cancelled; the run has no overall pass. Its concurrent update
finding is repaired. [Next CI
43909197530](https://github.com/paperclipai/paperclip/actions/runs/43909197530)
also passed types, build and all browser shards, but several server jobs
were cancelled. Greptile and Paperclip Review gave that revision 5/5.
Its security scan found unbounded retained cache rows. The quota,
retention and request-limit repair above addresses that finding. All ten
review threads are resolved.
- [Current CI
43912947278](https://github.com/paperclipai/paperclip/actions/runs/43912947278),
fresh code reviews and a fresh security scan are pending on
`335d89ef1668dcdeb66bb2059512a7c2da29e273`. Auto-merge is disabled until
the current revision clears its gates.

Published head: `335d89ef1668dcdeb66bb2059512a7c2da29e273`. Local head:
`b37b87c704a2ba8a65233c689a6a743f4011ce14`. Both have tree
`643ebd19b9729b1e6a6bb695442269c3db32be3a`; GitHub tree creation
verified equality.

## Risks

- Each company/viewer retains at most 1,000 rows. Writes remove excess
rows and entries whose freshness expired over 30 days ago. Inactive
scopes remain bounded and are cleaned on the next write. Company
deletion removes all rows. No periodic worker runs. Scope writes use a
transaction lock; the existing deadline and concurrency budget cover
that work.
- Last known status can be stale until idle or hover work completes.
Provider failures keep that status through the negative-cache window.
- Automatic enrichment adds bounded provider traffic. It runs only for
mounted supported links, once per mount, with the existing timeout and
server concurrency limit.
- Cache storage is viewer-scoped because provider results can be
private. New viewers load their own results. This prevents reuse across
users with different repository access.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools, code
execution and browser automation. The runtime does not expose the exact
serving model ID or context-window size. No subagents were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-11-16 14:33:39 -06:00
1 parent 0e8447fe77
commit f33c4cf6f7
27 files changed
+1572 -65

No files matched your search

@@ -441,10 +441,19 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
const [row] = await state.db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, watchdog.watchdog.id));
expect(row.watchdogIssueId).toBeTruthy(); reviewId = row.watchdogIssueId!;
}, { timeout: 15_000 });
await completed(reviewId!);
const review = await completed(reviewId!);
expect(failures).toEqual([]);
// Ordinary assignment recovery can win the wake reason while the watchdog
// wake coalesces into that run. Verify its mandate and one dispatch instead.
expect(review.contextSnapshot).toMatchObject({
watchdogId: watchdog.watchdog.id,
taskWatchdog: {
watchedIssueId: source.id,
stopFingerprint: expect.stringMatching(/^task_watchdog_stop:/),
},
});
expect(await state.db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'task_watchdog_stopped_subtree'`))).toHaveLength(1);
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${reviewId}`))).toHaveLength(1);
await finish(reviewId!);
await taskWatchdogService(state.db).disableForIssue(f.companyId, source.id);
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
@@ -4,21 +4,44 @@ import { describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
import { linkPreviewRoutes } from "../routes/link-previews.js";
import { HttpError } from "../errors.js";
import { createInviteRateLimiter, type InviteRateLimiter } from "../services/invite-rate-limit.js";
function app(actor: express.Request["actor"]) {
function app(actor: express.Request["actor"], limiter?: InviteRateLimiter) {
const get = vi.fn(async () => ({ preview: null, maxAgeMs: 60_000 }));
const cached = vi.fn(async () => ({ results: {} }));
const server = express();
server.use(express.json());
server.use((req, _res, next) => { req.actor = actor; next(); });
server.use(linkPreviewRoutes(null as unknown as Db, { get }));
server.use(linkPreviewRoutes(null as unknown as Db, { get, cached }, limiter));
server.use((error: HttpError, _req: express.Request, res: express.Response, _next: express.NextFunction) => {
res.status(error.status ?? 500).json({ error: error.message });
});
return { server, get };
return { server, get, cached };
}
const board = { type: "board", source: "session", userId: "alice", companyIds: ["company-a"] } as const;
const path = "/companies/company-a/link-preview";
describe("link preview routes", () => {
it("limits lookups per viewer and company without blocking cached reads", async () => {
let now = 0;
const limiter = createInviteRateLimiter({ maxRequests: 2, windowMs: 1_000, now: () => now });
const actor = { ...board, userId: "alice", companyIds: ["company-a", "company-b"] };
const { server, get, cached } = app(actor, limiter);
const query = { url: "https://github.com/acme/web/pull/42" };
for (let n = 0; n < 2; n++) expect((await request(server).get(path).query(query)).status).toBe(200);
const limited = await request(server).get(path).query(query);
expect(limited.status).toBe(429);
expect(limited.headers["retry-after"]).toBe("1");
expect(get).toHaveBeenCalledTimes(2);
expect((await request(server).get("/companies/company-a/link-previews/cached").query(query)).status).toBe(200);
expect(cached).toHaveBeenCalledTimes(1);
expect((await request(server).get("/companies/company-b/link-preview").query(query)).status).toBe(200);
actor.userId = "bob";
expect((await request(server).get(path).query(query)).status).toBe(200);
actor.userId = "alice";
now = 1_001;
expect((await request(server).get(path).query(query)).status).toBe(200);
});
it("passes the authenticated viewer and disables shared HTTP caching", async () => {
const { server, get } = app({ ...board, companyIds: [...board.companyIds] });
const response = await request(server).get(path).query({ url: "https://github.com/acme/web/pull/42" });
@@ -38,6 +61,25 @@ describe("link preview routes", () => {
expect(get).not.toHaveBeenCalled();
}
});
it("reads a bounded batch only after viewer and company authorization", async () => {
const { server, cached, get } = app({ ...board, companyIds: [...board.companyIds] });
const response = await request(server).get("/companies/company-a/link-previews/cached").query({ url: ["https://github.com/acme/web/pull/42"] });
expect(response.status).toBe(200);
expect(response.headers["cache-control"]).toBe("private, no-store");
expect(cached).toHaveBeenCalledWith({ companyId: "company-a", userId: "alice" }, ["https://github.com/acme/web/pull/42"]);
expect(get).not.toHaveBeenCalled();
for (const body of [{}, { url: ["x".repeat(2049)] }, { url: Array(51).fill("a") }]) {
expect((await request(server).get("/companies/company-a/link-previews/cached").query(body)).status).toBe(400);
}
const denied = app({ ...board, companyIds: ["company-b"] });
expect((await request(denied.server).get("/companies/company-a/link-previews/cached")).status).toBe(403);
expect(denied.cached).not.toHaveBeenCalled();
});
it("lets active viewer-role members read stored statuses", async () => {
const { server, cached } = app({ ...board, companyIds: [...board.companyIds], memberships: [{ companyId: "company-a", membershipRole: "viewer", status: "active" }] });
expect((await request(server).get("/companies/company-a/link-previews/cached").query({ url: "https://github.com/acme/web/pull/42" })).status).toBe(200);
expect(cached).toHaveBeenCalledTimes(1);
});
it("bounds and validates the URL input", async () => {
const { server, get } = app({ ...board, companyIds: [...board.companyIds] });
for (const query of [{}, { url: "x".repeat(2049) }, { url: ["a", "b"] }]) {
@@ -0,0 +1,253 @@
import { randomUUID } from "node:crypto";
import { readFile } from "node:fs/promises";
import { companies, linkPreviews } from "@paperclipai/db";
import { recognizeLinkPreview } from "@paperclipai/shared";
import { sql } from "drizzle-orm";
import { expect, it, vi } from "vitest";
import { describeEmbeddedPostgres, useEmbeddedPostgres } from "./helpers/route-test-harness.js";
import { createLinkPreviewStore } from "../services/link-preview-store.js";
import { createLinkPreviewService } from "../services/link-previews.js";
const url = "https://github.com/acme/web/pull/42";
const target = recognizeLinkPreview(url)!;
const preview = { target, title: "Durable PR", state: "merged" as const, author: "alice", updatedAt: null };
describeEmbeddedPostgres("durable link preview cache", () => {
const context = useEmbeddedPostgres("link-previews-");
async function fixture() {
const companyId = randomUUID();
await context.db.insert(companies).values({ id: companyId, name: "Link cache", issuePrefix: companyId });
return { companyId, userId: "alice" };
}
it("bounds retained rows across concurrent store instances and isolates their quotas", async () => {
const scope = await fixture();
const other = { ...scope, userId: "bob" };
const now = Date.now();
const seed = Array.from({ length: 1_000 }, (_, n) => ({
companyId: scope.companyId, viewerId: scope.userId, channel: "github", resourceKind: "pull_request",
canonicalUrl: url + n, status: null, metadata: {}, checkedAt: new Date(now - 1_000 + n), expiresAt: new Date(now + 60_000),
}));
await context.db.insert(linkPreviews).values([...seed, { ...seed[0], viewerId: other.userId }]);
const first = createLinkPreviewStore(context.db);
const second = createLinkPreviewStore(context.db);
await Promise.all(Array.from({ length: 8 }, (_, n) => (n % 2 ? first : second).write(scope,
recognizeLinkPreview(url + (1_000 + n))!, { result: { preview: null, maxAgeMs: 60_000 }, expiresAt: now + 60_000 }, now + n)));
const rows = await context.db.select().from(linkPreviews).where(sql`${linkPreviews.companyId} = ${scope.companyId}`);
expect(rows.filter(row => row.viewerId === scope.userId)).toHaveLength(1_000);
expect(rows.filter(row => row.viewerId === other.userId)).toHaveLength(1);
const saved = await first.read(scope, [recognizeLinkPreview(url + "0")!, recognizeLinkPreview(url + "1007")!]);
expect(saved.has(url + "0")).toBe(false);
expect(saved.has(url + "1007")).toBe(true);
});
it("removes long-expired rows on writes and preserves recent stale status", async () => {
const scope = await fixture();
const now = Date.now();
const expired = now - 31 * 24 * 60 * 60_000;
await context.db.insert(linkPreviews).values([
{ companyId: scope.companyId, viewerId: scope.userId, channel: "github", resourceKind: "pull_request", canonicalUrl: url,
status: "merged", metadata: { title: preview.title }, checkedAt: new Date(expired - 60_000), expiresAt: new Date(expired) },
{ companyId: scope.companyId, viewerId: scope.userId, channel: "github", resourceKind: "pull_request", canonicalUrl: url + "1",
status: "merged", metadata: { title: preview.title }, checkedAt: new Date(now - 120_000), expiresAt: new Date(now - 60_000) },
]);
const store = createLinkPreviewStore(context.db);
await store.write(scope, recognizeLinkPreview(url + "2")!, { result: { preview: null, maxAgeMs: 60_000 }, expiresAt: now + 60_000 }, now);
const saved = await store.read(scope, [target, recognizeLinkPreview(url + "1")!, recognizeLinkPreview(url + "2")!]);
expect(saved.has(url)).toBe(false);
expect(saved.get(url + "1")?.result.preview?.state).toBe("merged");
expect(saved.has(url + "2")).toBe(true);
});
it.each(["failure", "success", "timeout", "stale timeout"])("keeps another replica's concurrent hydration after provider %s", async (settlement) => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
const oldPreview = { ...preview, state: "open" as const };
await store.write(scope, target, { result: { preview: oldPreview, maxAgeMs: 1_000 }, expiresAt: 2_000 }, now);
let finish!: (value: { preview: typeof oldPreview | null; maxAgeMs: number }) => void;
let started!: () => void;
const providerStarted = new Promise<void>(done => { started = done; });
const resolve = vi.fn(() => new Promise<{ preview: typeof oldPreview | null; maxAgeMs: number }>(done => { finish = done; started(); }));
const write = vi.fn(store.write);
const first = createLinkPreviewService([{ key: "github", resolve }], { store: { ...store, write }, now: () => now, timeoutMs: 1_000 });
await first.cached(scope, [url]);
now = 3_000;
const pending = first.get(scope, url);
await providerStarted;
now = 4_000;
const expiresAt = settlement === "stale timeout" ? 3_500 : 34_000;
await store.write(scope, target, { result: { preview, maxAgeMs: 30_000 }, expiresAt }, now);
expect((await first.cached(scope, [url])).results[url].preview).toEqual(preview);
if (!settlement.includes("timeout")) finish({ preview: settlement === "success" ? oldPreview : null, maxAgeMs: 60_000 });
expect(await pending).toEqual({ preview, maxAgeMs: Math.max(0, expiresAt - now) });
// A second batch checks that the process did not retain the older status.
expect((await first.cached(scope, [url])).results[url].preview).toEqual(preview);
const saved = (await store.read(scope, [target])).get(url)!;
expect(saved.result.preview).toEqual(preview);
expect(saved.checkedAt).toBe(4_000);
expect(saved.expiresAt).toBe(expiresAt);
expect(write).not.toHaveBeenCalled();
finish({ preview: null, maxAgeMs: 60_000 });
});
it.each(["open", "negative"])("rechecks expired %s memory before calling the provider", async (initialState) => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
const resolve = vi.fn(async () => ({ preview: initialState === "open" ? { ...preview, state: "open" as const } : null, maxAgeMs: 60_000 }));
const first = createLinkPreviewService([{ key: "github", resolve }], { store, now: () => now });
await first.get(scope, url);
now = 62_000;
const other = createLinkPreviewService([{ key: "github", resolve: async () => ({ preview, maxAgeMs: 60_000 }) }], { store, now: () => now });
await other.get(scope, url);
resolve.mockRejectedValue(new Error("provider unavailable"));
expect((await first.get(scope, url)).preview).toEqual(preview);
expect(resolve).toHaveBeenCalledTimes(1);
const saved = (await store.read(scope, [target])).get(url)!;
expect(saved.result.preview).toEqual(preview);
expect(saved.expiresAt).toBe(122_000);
});
it.each(["failure", "timeout"])("retains the newest stale stored preview on provider %s", async (failure) => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
let requests = 0;
const resolve = async (_target: import("@paperclipai/shared").LinkPreviewTarget, _scope: import("../services/link-previews.js").LinkPreviewScope, signal: AbortSignal) => {
if (++requests === 1) return { preview: { ...preview, state: "open" as const }, maxAgeMs: 60_000 };
if (failure === "failure") throw new Error("provider unavailable");
return new Promise<never>((_done, reject) => signal.addEventListener("abort", () => reject(new Error("timeout"))));
};
const first = createLinkPreviewService([{ key: "github", resolve }], { store, now: () => now, timeoutMs: 100 });
await first.get(scope, url);
now = 62_000;
await store.write(scope, target, { result: { preview, maxAgeMs: 1_000 }, expiresAt: 63_000 }, now);
now = 64_000;
expect((await first.get(scope, url)).preview).toEqual(preview);
await vi.waitFor(async () => expect((await store.read(scope, [target])).get(url)?.expiresAt).toBe(124_000));
expect((await store.read(scope, [target])).get(url)?.result.preview).toEqual(preview);
});
it("compares check times rather than cache lifetimes when hydrating", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
await store.write(scope, target, { result: { preview: { ...preview, state: "open" }, maxAgeMs: 60_000 }, expiresAt: 61_000 }, now);
const service = createLinkPreviewService([{ key: "github", resolve: async () => ({ preview: null, maxAgeMs: 60_000 }) }], { store, now: () => now });
await service.cached(scope, [url]);
await store.write(scope, target, { result: { preview, maxAgeMs: 1_000 }, expiresAt: 4_000 }, 3_000);
now = 5_000;
expect((await service.cached(scope, [url])).results[url]).toEqual({ preview, maxAgeMs: 0 });
});
it("keeps a failed durable read's fallback local and accepts recovered stored status", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const read = vi.fn(store.read);
let now = 1_000;
const resolve = vi.fn(async () => ({ preview: { ...preview, state: "open" as const }, maxAgeMs: 60_000 }));
const first = createLinkPreviewService([{ key: "github", resolve }], { store: { ...store, read }, now: () => now });
await first.get(scope, url);
now = 62_000;
await store.write(scope, target, { result: { preview, maxAgeMs: 60_000 }, expiresAt: 122_000 }, now);
now = 64_000;
read.mockRejectedValueOnce(new Error("temporary database outage"));
resolve.mockRejectedValueOnce(new Error("provider unavailable"));
expect((await first.get(scope, url)).preview?.state).toBe("open");
const saved = (await store.read(scope, [target])).get(url)!;
expect(saved.result.preview).toEqual(preview);
expect(saved.expiresAt).toBe(122_000);
expect((await first.cached(scope, [url])).results[url].preview).toEqual(preview);
});
it.each(["missing", "failed"])("reads another replica's saved status after a %s batch read", async (initialRead) => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const read = vi.fn(store.read);
if (initialRead === "failed") read.mockRejectedValueOnce(new Error("temporary database outage"));
const write = vi.fn(store.write);
const resolve = vi.fn(async () => { throw new Error("provider unavailable"); });
const first = createLinkPreviewService([{ key: "github", resolve }], { store: { read, write } });
expect(await first.cached(scope, [url])).toEqual({ results: {} });
const other = createLinkPreviewService([{ key: "github", resolve: async () => ({ preview, maxAgeMs: 60_000 }) }], { store });
await other.get(scope, url);
expect((await first.get(scope, url)).preview).toEqual(preview);
expect(read).toHaveBeenCalledTimes(2);
expect(resolve).not.toHaveBeenCalled();
expect(write).not.toHaveBeenCalled();
expect((await store.read(scope, [target])).get(url)?.result.preview).toEqual(preview);
});
it("survives service recreation, reuses canonical URLs across tasks, and isolates viewers/companies", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const resolve = vi.fn(async () => ({ preview, maxAgeMs: 60_000 }));
const make = () => createLinkPreviewService([{ key: "github", resolve }], { store });
await make().get(scope, url);
const restored = await make().cached(scope, [url, "https://www.github.com/ACME/WEB/pull/42/files#review"]);
expect(Object.keys(restored.results)).toEqual([url]);
expect(restored.results[url].preview).toMatchObject({ title: preview.title, state: "merged", target: { canonicalUrl: url } });
expect(restored.results[url].maxAgeMs).toBeGreaterThan(0);
await make().get(scope, url);
expect(resolve).toHaveBeenCalledTimes(1);
expect(await make().cached({ ...scope, userId: "bob" }, [url])).toEqual({ results: {} });
expect(await make().cached(await fixture(), [url])).toEqual({ results: {} });
});
it("returns stale status immediately, refreshes on demand, and persists negative freshness", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
const resolve = vi.fn(async () => ({ preview, maxAgeMs: 60_000 }));
const make = () => createLinkPreviewService([{ key: "github", resolve }], { store, now: () => now });
await make().get(scope, url);
now += 61_000;
expect((await make().cached(scope, [url])).results[url]).toEqual({ preview, maxAgeMs: 0 });
expect(resolve).toHaveBeenCalledTimes(1);
resolve.mockRejectedValueOnce(new Error("outage"));
expect((await make().get(scope, url)).preview).toEqual(preview);
await make().get(scope, url);
expect(resolve).toHaveBeenCalledTimes(2);
const missing = url + "1";
resolve.mockRejectedValueOnce(new Error("private"));
await make().get(scope, missing);
await make().get(scope, missing);
expect(resolve).toHaveBeenCalledTimes(3);
expect((await make().cached(scope, [missing])).results[missing].preview).toBeNull();
});
it("persists provider timeout freshness across service restarts without extending the response", async () => {
const scope = await fixture();
const database = createLinkPreviewStore(context.db);
let saved!: () => void;
const persisted = new Promise<void>(done => { saved = done; });
const store = { ...database, write: async (...args: Parameters<typeof database.write>) => { await database.write(...args); saved(); } };
const resolve = vi.fn((_target: import("@paperclipai/shared").LinkPreviewTarget, _scope: import("../services/link-previews.js").LinkPreviewScope, signal: AbortSignal) => new Promise<never>((_done, reject) => {
signal.addEventListener("abort", () => reject(new Error("timeout")));
}));
const first = createLinkPreviewService([{ key: "github", resolve }], { store, timeoutMs: 100 });
expect((await first.get(scope, url)).preview).toBeNull();
await persisted;
const restarted = createLinkPreviewService([{ key: "github", resolve }], { store });
expect((await restarted.get(scope, url)).preview).toBeNull();
expect(resolve).toHaveBeenCalledTimes(1);
expect((await restarted.cached(scope, [url])).results[url].maxAgeMs).toBeGreaterThan(0);
});
it("reads a 50-link page in one indexed select and does not require task relationships", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const targets = Array.from({ length: 50 }, (_, n) => recognizeLinkPreview(url + n)!);
await Promise.all(targets.map(t => store.write(scope, t, { result: { preview: { ...preview, target: t }, maxAgeMs: 60_000 }, expiresAt: 61_000 }, 1_000)));
const select = vi.spyOn(context.db, "select");
expect((await store.read(scope, targets)).size).toBe(50);
expect(select).toHaveBeenCalledTimes(1);
select.mockRestore();
const plan = await context.db.execute(sql`EXPLAIN SELECT * FROM link_previews WHERE company_id = ${scope.companyId} AND viewer_id = 'alice' AND canonical_url IN (${sql.join(targets.map(t => sql`${t.canonicalUrl}`), sql`, `)})`);
expect(JSON.stringify(plan)).toContain("link_previews_scope_url_uq");
});
it("supports other channels/statuses and older writers cannot replace newer results", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const deployment = { ...target, provider: "vercel", kind: "deployment", canonicalUrl: "https://vercel.com/acme/build/123" };
await context.db.insert(linkPreviews).values({ companyId: scope.companyId, viewerId: scope.userId, channel: deployment.provider,
resourceKind: deployment.kind, canonicalUrl: deployment.canonicalUrl, status: "deployed", metadata: { deploymentId: "123", region: "iad1" }, checkedAt: new Date(), expiresAt: new Date() });
await store.write(scope, target, { result: { preview, maxAgeMs: 60_000 }, expiresAt: 65_000 }, 5_000);
await store.write(scope, target, { result: { preview: { ...preview, state: "open" }, maxAgeMs: 60_000 }, expiresAt: 61_000 }, 1_000);
expect((await store.read(scope, [target])).get(url)?.result.preview?.state).toBe("merged");
const migration = await readFile(new URL("../../../packages/db/src/migrations/0489_real_sage.sql", import.meta.url), "utf8");
for (const statement of migration.split("--> statement-breakpoint")) if (statement.trim()) await context.db.execute(sql.raw(statement));
expect((await store.read(scope, [target])).get(url)?.result.preview?.state).toBe("merged");
});
});
@@ -2,6 +2,8 @@ import { createServer } from "node:http";
import { once } from "node:events";
import { afterEach, describe, expect, it, vi } from "vitest";
import { createLinkPreviewService, type LinkPreviewProvider } from "../services/link-previews.js";
import type { StoredLinkPreview } from "../services/link-preview-store.js";
import { recognizeLinkPreview } from "@paperclipai/shared";
const scope = { companyId: "company-a", userId: "alice" };
const url = "https://github.com/acme/web/pull/42";
@@ -10,6 +12,51 @@ const provider = (resolve = vi.fn(async () => unavailable)): LinkPreviewProvider
afterEach(() => vi.useRealTimers());
describe("demand-driven link preview cache", () => {
it.each(["failure", "success", "timeout", "write"])("keeps concurrent hydration newer than a pending provider %s", async (settlement) => {
vi.useFakeTimers();
let now = 1_000;
const target = recognizeLinkPreview(url)!;
const oldPreview = { target, title: "Old status", state: "open" as const, author: null, updatedAt: null };
const newPreview = { ...oldPreview, title: "New status", state: "merged" as const };
let stored: StoredLinkPreview = { result: { preview: oldPreview, maxAgeMs: 1_000 }, expiresAt: 2_000, checkedAt: now };
let finish!: (value: typeof unavailable | { preview: typeof oldPreview; maxAgeMs: number }) => void;
let finishWrite!: () => void;
const write = vi.fn(async () => {
if (settlement === "write") await new Promise<void>(done => { finishWrite = done; });
});
const resolve = vi.fn(() => new Promise<typeof unavailable | { preview: typeof oldPreview; maxAgeMs: number }>(done => { finish = done; }));
const service = createLinkPreviewService([{ key: "github", resolve }], {
now: () => now, timeoutMs: 20,
store: { read: async () => new Map([[url, stored]]), write },
});
await service.cached(scope, [url]);
now = 3_000;
const pending = service.get(scope, url);
await vi.advanceTimersByTimeAsync(0);
if (settlement === "write") {
finish({ preview: oldPreview, maxAgeMs: 60_000 });
await vi.advanceTimersByTimeAsync(0);
expect(write).toHaveBeenCalledTimes(1);
}
now = 4_000;
stored = { result: { preview: newPreview, maxAgeMs: 30_000 }, expiresAt: 34_000, checkedAt: now };
expect((await service.cached(scope, [url])).results[url].preview).toEqual(newPreview);
if (settlement === "write") finishWrite();
else if (settlement === "timeout") await vi.advanceTimersByTimeAsync(21);
else finish(settlement === "failure" ? unavailable : { preview: oldPreview, maxAgeMs: 60_000 });
expect(await pending).toEqual({ preview: newPreview, maxAgeMs: 30_000 });
expect(await service.get(scope, url)).toEqual({ preview: newPreview, maxAgeMs: 30_000 });
expect(resolve).toHaveBeenCalledTimes(1);
expect(write).toHaveBeenCalledTimes(settlement === "write" ? 1 : 0);
if (settlement === "timeout") { finish(unavailable); await vi.advanceTimersByTimeAsync(0); }
});
it("omits absent records on repeated batch reads", async () => {
const service = createLinkPreviewService([provider()], {
store: { read: async () => new Map(), write: async () => {} },
});
expect(await service.cached(scope, [url])).toEqual({ results: {} });
expect(await service.cached(scope, [url])).toEqual({ results: {} });
});
it("does no work until requested and coalesces equivalent URLs", async () => {
const resolve = vi.fn(async () => unavailable);
const service = createLinkPreviewService([provider(resolve)]);
@@ -86,6 +133,36 @@ describe("demand-driven link preview cache", () => {
expect(resolve).toHaveBeenCalledTimes(2);
finish(unavailable);
});
it("bounds a stalled database write and holds its concurrency slot until settlement", async () => {
vi.useFakeTimers();
let finish!: () => void;
const write = vi.fn(async () => {}).mockImplementationOnce(() => new Promise<void>(done => { finish = done; }));
const resolve = vi.fn(async (target: import("@paperclipai/shared").LinkPreviewTarget) => ({ preview: { target, title: "Known", state: "open" as const, author: null, updatedAt: null }, maxAgeMs: 60_000 }));
const service = createLinkPreviewService([{ key: "github", resolve }], {
store: { read: async () => new Map(), write }, timeoutMs: 20, maxConcurrent: 1,
});
const first = service.get(scope, url);
await vi.advanceTimersByTimeAsync(21);
expect((await first).preview?.state).toBe("open");
expect(await service.get(scope, url + "1")).toEqual(unavailable);
expect(resolve).toHaveBeenCalledTimes(1);
finish(); await vi.advanceTimersByTimeAsync(1);
await service.get(scope, url + "1");
expect(resolve).toHaveBeenCalledTimes(2);
});
it("keeps timeout persistence inside the concurrency budget after the provider aborts", async () => {
vi.useFakeTimers();
let finishWrite!: () => void;
const write = vi.fn(async () => {}).mockImplementationOnce(() => new Promise<void>(done => { finishWrite = done; }));
const resolve = vi.fn(async (_target: import("@paperclipai/shared").LinkPreviewTarget, _scope: import("../services/link-previews.js").LinkPreviewScope, signal: AbortSignal) => unavailable)
.mockImplementationOnce((_target, _scope, signal) => new Promise((_done, reject) => signal.addEventListener("abort", () => reject(new Error("timeout")))));
const service = createLinkPreviewService([{ key: "github", resolve }], { store: { read: async () => new Map(), write }, timeoutMs: 20, maxConcurrent: 1 });
const first = service.get(scope, url);
await vi.advanceTimersByTimeAsync(21); expect(await first).toEqual(unavailable);
await service.get(scope, url + "1"); expect(resolve).toHaveBeenCalledTimes(1);
finishWrite(); await vi.advanceTimersByTimeAsync(1);
await service.get(scope, url + "1"); expect(resolve).toHaveBeenCalledTimes(2);
});
it("aborts a real stalled HTTP read while an unrelated lookup remains responsive", async () => {
const server = createServer((_req, res) => { res.writeHead(200, { "content-type": "application/json" }); res.write("{"); });
server.listen(0, "127.0.0.1");
+10 -1
View File
@@ -4,7 +4,7 @@ import path from "node:path";
import { fileURLToPath } from "node:url";
import request from "supertest";
import { describe, expect, it } from "vitest";
import { MUSE_ASSET_VERSION, MUSE_PUBLIC_ROUTES } from "@paperclipai/shared";
import { LINK_PREVIEW_BATCH_LIMIT, MUSE_ASSET_VERSION, MUSE_PUBLIC_ROUTES } from "@paperclipai/shared";
import { COMPANY_IMPORT_TRANSFERS_ROUTE_PATH } from "@paperclipai/shared/company-import-transfer";
import { errorHandler } from "../middleware/index.js";
import { buildOpenApiSpec, openApiRoutes, applyPluginSettingsVisibility, buildOpenApiDocument } from "../routes/openapi.js";
@@ -338,6 +338,15 @@ function loadSpecRoutes() {
}
describe("openapi routes", () => {
it("describes cached link results with the lookup shape and shared batch bound", () => {
const paths = buildOpenApiSpec().paths;
const batch = paths["/api/companies/{companyId}/link-previews/cached"].get;
expect(batch.description).toContain(`at most ${LINK_PREVIEW_BATCH_LIMIT} URLs`);
const lookupResult = paths["/api/companies/{companyId}/link-preview"].get.responses[200].content["application/json"].schema;
expect(batch.responses[200].content["application/json"].schema.properties.results.additionalProperties).toEqual(lookupResult);
expect(lookupResult.properties).toHaveProperty("preview");
expect(lookupResult.properties).toHaveProperty("maxAgeMs");
});
it("tells agents that unpriced usage in cost reports is unknown cost, not zero", () => {
const paths = buildOpenApiSpec().paths;
for (const segment of ["summary", "by-agent", "by-agent-model", "by-project"]) {
+24 -2
View File
@@ -1,11 +1,15 @@
import { Router } from "express";
import type { Db } from "@paperclipai/db";
import { badRequest } from "../errors.js";
import { LINK_PREVIEW_BATCH_LIMIT } from "@paperclipai/shared";
import { createLinkPreviewStore } from "../services/link-preview-store.js";
import { badRequest, tooManyRequests } from "../errors.js";
import { createInviteRateLimiter, type InviteRateLimiter } from "../services/invite-rate-limit.js";
import { assertBoard, assertCompanyAccess } from "./authz.js";
import { createLinkPreviewService } from "../services/link-previews.js";
import { createGitHubLinkPreviewProvider } from "../services/github-link-preview.js";
export function linkPreviewRoutes(db: Db, service = createLinkPreviewService([createGitHubLinkPreviewProvider(db)])) {
export function linkPreviewRoutes(db: Db, service = createLinkPreviewService([createGitHubLinkPreviewProvider(db)], { store: createLinkPreviewStore(db) }),
limiter: InviteRateLimiter = createInviteRateLimiter({ maxRequests: 600 })) {
const router = Router();
router.get("/companies/:companyId/link-preview", async (req, res) => {
const companyId = req.params.companyId as string;
@@ -15,7 +19,25 @@ export function linkPreviewRoutes(db: Db, service = createLinkPreviewService([cr
if (typeof href !== "string" || href.length > 2048) throw badRequest("A link URL is required");
// Personal repository metadata must never be stored by a proxy or shared browser cache.
res.set("Cache-Control", "private, no-store");
const limit = limiter.consume(JSON.stringify([companyId, req.actor.userId ?? null]));
res.set("X-RateLimit-Limit", String(limit.limit));
res.set("X-RateLimit-Remaining", String(limit.remaining));
if (!limit.allowed) {
res.set("Retry-After", String(limit.retryAfterSeconds));
throw tooManyRequests("Link preview rate limit exceeded");
}
res.json(await service.get({ companyId, userId: req.actor.userId ?? null }, href));
});
// Read-only batch: page hydration never waits for a remote provider.
router.get("/companies/:companyId/link-previews/cached", async (req, res) => {
const companyId = req.params.companyId as string;
assertCompanyAccess(req, companyId);
assertBoard(req);
const urls = typeof req.query.url === "string" ? [req.query.url] : req.query.url;
if (!Array.isArray(urls) || !urls.length || urls.length > LINK_PREVIEW_BATCH_LIMIT
|| urls.some(url => typeof url !== "string" || url.length > 2048)) throw badRequest("A bounded list of link URLs is required");
res.set("Cache-Control", "private, no-store");
res.json(await service.cached({ companyId, userId: req.actor.userId ?? null }, urls as string[]));
});
return router;
}
+26 -11
View File
@@ -2,7 +2,7 @@ import { catalogTeamInstallSchema, catalogTeamPreviewSchema, updateWorkspacePref
import { MUSE_ASSET_VERSION, museStopBoundarySchema, museDetectorCleanupAttestationSchema, createGrokInvitationSchema, configureGrokRoutineSchema, verifyGrokRoutineSchema, recoverGrokRoutineSchema, pairGrokBindingSchema, stopGrokWorkSchema, retryGrokDeliverySchema, updateFastResponseSchema } from "@paperclipai/shared";
import { slackTestMessageSchema, slackTestMessageResultSchema, slackManagerAuthorizationSchema, slackSetupOptionsSchema, slackManagedProvisionSchema, slackManagedResultSchema, slackRegistrationSchema, slackSetupActionSchema, slackInstallAuthorizationSchema, slackRegistrationStateSchema, slackAppConfigurationSchema, slackAvatarStateSchema, slackAccountStateSchema } from "@paperclipai/shared";
import { experimentalApiMetadata } from "./experimental-api-metadata.js";
import { ATTENTION_SOURCE_KINDS } from "@paperclipai/shared";
import { LINK_PREVIEW_BATCH_LIMIT, ATTENTION_SOURCE_KINDS } from "@paperclipai/shared";
import { agentCaseHandoffSchema, caseWorkflowDocumentTextSchema, doneCaseHandoffSchema, createWorkflowSchema, updateWorkflowSchema, listWorkflowsSchema, readWorkflowQuerySchema } from "@paperclipai/shared";
import {
experimentalApiDocs,
@@ -10459,26 +10459,41 @@ registry.registerPath({
responses: { 200: r.ok(), 401: r.unauthorized, 404: r.notFound },
});
const linkPreviewResultSchema = z.object({
preview: z.object({
target: z.object({ provider: z.string(), kind: z.string(), canonicalUrl: z.string(), label: z.string() }),
title: z.string(), state: z.enum(["unknown", "open", "draft", "merged", "closed", "conflicts"]),
author: z.string().nullable(), updatedAt: z.string().nullable(),
}).nullable(),
maxAgeMs: z.number(),
});
registry.registerPath({
method: "get",
path: "/api/companies/{companyId}/link-preview",
tags: ["companies"],
summary: "Read optional link metadata for the current board user",
description: "Company access and board authentication are required. This read uses a viewer-scoped cache and returns a quiet null preview when provider access is unavailable. It does not change tasks or schedule background work.",
description: "Company access and board authentication are required. This read uses a viewer-scoped cache and returns a quiet null preview when provider access is unavailable. Each company/viewer can make 600 lookup requests per minute per server process. Excess requests return 429 with Retry-After; cached batch reads remain available. It does not change tasks or schedule background work.",
request: { params: z.object({ companyId: z.string() }), query: z.object({ url: z.string().max(2048) }) },
responses: {
200: r.ok(z.object({
preview: z.object({
target: z.object({ provider: z.string(), kind: z.string(), canonicalUrl: z.string(), label: z.string() }),
title: z.string(), state: z.enum(["unknown", "open", "draft", "merged", "closed", "conflicts"]),
author: z.string().nullable(), updatedAt: z.string().nullable(),
}).nullable(),
maxAgeMs: z.number(),
})),
400: r.badRequest, 401: r.unauthorized, 403: r.forbidden,
200: r.ok(linkPreviewResultSchema),
400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 429: r.tooManyRequests,
},
});
registry.registerPath({
method: "get",
path: "/api/companies/{companyId}/link-previews/cached",
tags: ["companies"],
summary: "Read a batch of stored link statuses without contacting providers",
description: `Board-only and company/viewer scoped. Reads at most ${LINK_PREVIEW_BATCH_LIMIT} URLs. Canonical links are reused across tasks. Stale records have maxAgeMs zero; absent records are omitted. This is a read-only cache projection.`,
request: {
params: z.object({ companyId: z.string() }),
query: z.object({ url: z.union([z.string().max(2048), z.array(z.string().max(2048)).min(1).max(LINK_PREVIEW_BATCH_LIMIT)]) }),
},
responses: { 200: r.ok(z.object({ results: z.record(z.string(), linkPreviewResultSchema) })), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden },
});
registry.registerPath({
method: "get",
path: "/api/issues/{id}/external-objects",
+65
View File
@@ -0,0 +1,65 @@
import { and, desc, eq, inArray, lt, lte, or, sql } from "drizzle-orm";
import { linkPreviews, type Db } from "@paperclipai/db";
import type { LinkPreviewResult, LinkPreviewTarget } from "@paperclipai/shared";
import type { LinkPreviewScope } from "./link-previews.js";
export interface StoredLinkPreview { result: LinkPreviewResult; expiresAt: number; checkedAt: number }
export interface LinkPreviewStore {
read(scope: LinkPreviewScope, targets: readonly LinkPreviewTarget[]): Promise<Map<string, StoredLinkPreview>>;
write(scope: LinkPreviewScope, target: LinkPreviewTarget, entry: Omit<StoredLinkPreview, "checkedAt">, checkedAt: number): Promise<void>;
}
const MAX_ROWS_PER_VIEWER = 1_000;
const EXPIRED_RETENTION_MS = 30 * 24 * 60 * 60_000;
export function createLinkPreviewStore(db: Db): LinkPreviewStore {
return {
async read(scope, targets) {
if (!targets.length) return new Map();
// One covering identity-index lookup for the page, never a task join or a provider call.
const rows = await db.select().from(linkPreviews).where(and(
eq(linkPreviews.companyId, scope.companyId), eq(linkPreviews.viewerId, scope.userId ?? ""),
inArray(linkPreviews.canonicalUrl, targets.map(target => target.canonicalUrl)),
));
const byUrl = new Map(targets.map(target => [target.canonicalUrl, target]));
return new Map(rows.flatMap(row => {
const target = byUrl.get(row.canonicalUrl);
if (!target || target.provider !== row.channel || target.kind !== row.resourceKind) return [];
const data = row.metadata;
const preview = row.status !== null && typeof data.title === "string" ? {
target, title: data.title, state: row.status as NonNullable<LinkPreviewResult["preview"]>["state"],
author: typeof data.author === "string" ? data.author : null,
updatedAt: typeof data.updatedAt === "string" ? data.updatedAt : null,
} : null;
return [[row.canonicalUrl, { result: { preview, maxAgeMs: 0 }, expiresAt: row.expiresAt.getTime(), checkedAt: row.checkedAt.getTime() }] as const];
}));
},
async write(scope, target, entry, checkedAt) {
const preview = entry.result.preview;
const values = {
companyId: scope.companyId, viewerId: scope.userId ?? "", channel: target.provider,
resourceKind: target.kind, canonicalUrl: target.canonicalUrl, status: preview?.state ?? null,
metadata: preview ? { title: preview.title, author: preview.author, updatedAt: preview.updatedAt } : {},
checkedAt: new Date(checkedAt), expiresAt: new Date(entry.expiresAt),
};
await db.transaction(async tx => {
// Serialize only this cache scope, including across server replicas.
// The row cap must hold when several distinct URLs are saved together.
const lockKey = JSON.stringify(["link-previews", values.companyId, values.viewerId]);
await tx.execute(sql`SELECT pg_advisory_xact_lock(hashtextextended(${lockKey}, 0))`);
await tx.insert(linkPreviews).values(values).onConflictDoUpdate({
target: [linkPreviews.companyId, linkPreviews.viewerId, linkPreviews.canonicalUrl],
set: values, setWhere: lte(linkPreviews.checkedAt, values.checkedAt),
});
const scopeCondition = and(eq(linkPreviews.companyId, values.companyId), eq(linkPreviews.viewerId, values.viewerId));
// Cleanup runs on writes, without a worker or extra page-read queries.
// The identity index covers this scope, whose retained size is bounded.
await tx.delete(linkPreviews).where(and(scopeCondition, or(
lt(linkPreviews.expiresAt, new Date(checkedAt - EXPIRED_RETENTION_MS)),
inArray(linkPreviews.id, tx.select({ id: linkPreviews.id }).from(linkPreviews).where(scopeCondition)
.orderBy(desc(linkPreviews.checkedAt), desc(linkPreviews.id)).offset(MAX_ROWS_PER_VIEWER)),
)));
});
},
};
}
+96 -14
View File
@@ -1,4 +1,5 @@
import type { LinkPreviewResult, LinkPreviewTarget } from "@paperclipai/shared";
import type { LinkPreviewStore, StoredLinkPreview } from "./link-preview-store.js";
import { recognizeLinkPreview } from "@paperclipai/shared";
export interface LinkPreviewScope { companyId: string; userId: string | null }
@@ -9,22 +10,49 @@ export interface LinkPreviewProvider {
const UNAVAILABLE: LinkPreviewResult = { preview: null, maxAgeMs: 60_000 };
/** Request-driven and process-local. No DB writes, sweep, timers while idle, or cross-viewer reuse. */
/** Request-driven cache: durable across restarts, isolated by viewer, with no idle worker. */
export function createLinkPreviewService(providers: readonly LinkPreviewProvider[], options: {
now?: () => number; timeoutMs?: number; maxEntries?: number; maxConcurrent?: number;
store?: LinkPreviewStore; now?: () => number; timeoutMs?: number; maxEntries?: number; maxConcurrent?: number;
} = {}) {
const now = options.now ?? Date.now;
const cache = new Map<string, { result: LinkPreviewResult; expiresAt: number }>();
const cache = new Map<string, StoredLinkPreview>();
const inFlight = new Map<string, Promise<LinkPreviewResult>>();
let activeOperations = 0;
const byKey = new Map(providers.map(provider => [provider.key, provider]));
function remember(key: string, entry: StoredLinkPreview) {
if ((cache.get(key)?.checkedAt ?? -Infinity) > entry.checkedAt) return;
cache.delete(key);
while (cache.size >= (options.maxEntries ?? 1_000)) cache.delete(cache.keys().next().value!);
cache.set(key, entry);
}
const keyOf = (scope: LinkPreviewScope, url: string) => JSON.stringify([scope.companyId, scope.userId, url]);
const remaining = (entry: StoredLinkPreview) => ({ ...entry.result, maxAgeMs: Math.max(0, entry.expiresAt - now()) });
const newest = (memory: StoredLinkPreview | undefined, stored: StoredLinkPreview | undefined) =>
stored && (!memory || stored.checkedAt >= memory.checkedAt) ? stored : memory;
return {
async cached(scope: LinkPreviewScope, hrefs: readonly string[]) {
const targets = [...new Map(hrefs.map(recognizeLinkPreview).filter(target => target && byKey.has(target.provider))
.map(target => [target!.canonicalUrl, target!])).values()];
const stored = await options.store?.read(scope, targets).catch(() => new Map<string, StoredLinkPreview>());
const results: Record<string, LinkPreviewResult> = {};
for (const target of targets) {
const key = keyOf(scope, target.canonicalUrl);
const memory = cache.get(key);
const entry = newest(memory, stored?.get(target.canonicalUrl));
if (entry) { remember(key, entry); results[target.canonicalUrl] = remaining(entry); }
// A batch miss is not a provider result. A later lookup must still
// check durable storage, which another replica may have populated.
else cache.delete(key);
}
return { results };
},
async get(scope: LinkPreviewScope, href: string): Promise<LinkPreviewResult> {
const target = recognizeLinkPreview(href);
const provider = target && byKey.get(target.provider);
if (!target || !provider) return UNAVAILABLE;
const key = JSON.stringify([scope.companyId, scope.userId, target.canonicalUrl]);
const key = keyOf(scope, target.canonicalUrl);
const cached = cache.get(key);
if (cached && cached.expiresAt > now()) {
return { ...cached.result, maxAgeMs: cached.expiresAt - now() };
@@ -35,20 +63,74 @@ export function createLinkPreviewService(providers: readonly LinkPreviewProvider
const controller = new AbortController();
let timer: ReturnType<typeof setTimeout>;
let previous = cached;
const checkedAt = now();
let providerStarted = false;
let storeWriteStarted = false;
let durableReadSucceeded = !options.store;
let timeoutWrite: Promise<void> | undefined;
const fallback = (): LinkPreviewResult => ({ preview: newest(previous, cache.get(key))?.result.preview ?? null, maxAgeMs: 60_000 });
const deadline = new Promise<LinkPreviewResult>(resolve => {
timer = setTimeout(() => { controller.abort(); resolve(UNAVAILABLE); }, options.timeoutMs ?? 5_000);
timer = setTimeout(() => {
const current = cache.get(key);
if (current && current.checkedAt > checkedAt) {
controller.abort();
resolve(remaining(current));
return;
}
const result = current && current.expiresAt > now() ? remaining(current) : fallback();
const entry = { result, expiresAt: now() + result.maxAgeMs,
checkedAt: durableReadSucceeded || storeWriteStarted ? checkedAt : previous?.checkedAt ?? 0 };
remember(key, entry);
// Publish timeout freshness without delaying the caller. The lookup
// keeps its slot until both the provider and this write settle.
if (options.store && durableReadSucceeded && providerStarted && !storeWriteStarted) {
timeoutWrite = Promise.resolve().then(() => options.store!.write(scope, target, entry, checkedAt)).catch(() => {});
}
controller.abort();
resolve(result);
}, options.timeoutMs ?? 5_000);
});
// Keep an underlying operation in the concurrency budget until it settles,
// even if a credential backend ignores cancellation after our deadline.
// All underlying work, including uncancellable SQL, retains its slot until
// settlement. The caller's deadline does not release the resource budget.
activeOperations++;
const operation = Promise.resolve().then(() => provider.resolve(target, scope, controller.signal))
.catch(() => UNAVAILABLE).finally(() => { activeOperations--; });
const request = Promise.race([operation, deadline]).then(result => {
const operation = Promise.resolve().then(async () => {
if (options.store) {
// Fresh memory returned above. Expired memory is only a fallback:
// another replica may have saved a newer result, even with a shorter TTL.
const stored = await options.store.read(scope, [target]).catch(() => null);
durableReadSucceeded = stored !== null;
previous = newest(newest(previous, cache.get(key)), stored?.get(target.canonicalUrl));
if (!controller.signal.aborted && previous) remember(key, previous);
}
if (controller.signal.aborted) return fallback();
if (previous && previous.expiresAt > now()) { remember(key, previous); return remaining(previous); }
providerStarted = true;
const result = await provider.resolve(target, scope, controller.signal).catch(() => UNAVAILABLE);
if (controller.signal.aborted) return fallback();
// A page batch can hydrate another replica's newer status while the
// provider is pending. That result wins over this older lookup.
previous = newest(previous, cache.get(key));
if (previous && previous.checkedAt > checkedAt) return remaining(previous);
const bounded = { ...result, maxAgeMs: Math.max(1_000, Math.min(result.maxAgeMs, 60 * 60_000)) };
cache.delete(key);
while (cache.size >= (options.maxEntries ?? 1_000)) cache.delete(cache.keys().next().value!);
cache.set(key, { result: bounded, expiresAt: now() + bounded.maxAgeMs });
return bounded;
if (!bounded.preview && previous?.result.preview) bounded.preview = previous.result.preview;
const entry = { result: bounded, expiresAt: now() + bounded.maxAgeMs,
checkedAt: durableReadSucceeded || result.preview ? checkedAt : previous?.checkedAt ?? 0 };
remember(key, entry);
// A failed durable read cannot establish which saved status a provider
// failure would replace. Keep that backoff local until storage recovers.
if (options.store && (durableReadSucceeded || result.preview)) {
storeWriteStarted = true;
await options.store.write(scope, target, entry, checkedAt).catch(() => {});
}
return remaining(entry);
}).catch(fallback).finally(async () => { await timeoutWrite; activeOperations--; });
const request = Promise.race([operation, deadline]).then(result => {
const current = cache.get(key);
// Hydration can also finish while the operation awaits its SQL write.
if (current && current.checkedAt > checkedAt) return remaining(current);
if (!current || current.expiresAt <= now()) remember(key, { result, expiresAt: now() + result.maxAgeMs, checkedAt });
return result;
}).finally(() => {
clearTimeout(timer!);
if (inFlight.get(key) === request) inFlight.delete(key);