feat(links): cache rich link status in the database and load it on idle (#21746)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task text contains links to pull requests and other work.
> - A link has useful status that should be visible when a task opens.
> - The current cache lives in the server process and the browser.
> - A new browser or server restart has no stored status to show.
> - This pull request stores link status in the database and reads it in
batches.
> - Idle page work updates stale status without a hover or a delay to
task data.

## Linked Issues or Issue Description

**Problem or motivation**

PR status is unknown in a new browser until the user points to each
link. The process cache cannot supply saved status after a server
restart. Related work: #20593 and #20654. A search found no other PR for
database link caching.

**Proposed solution**

Store status by company, viewer and canonical resource URL. Read saved
status in one indexed batch query. After document load and core request
settlement, use browser idle time to update missing or stale status. Use
the same query for hover and idle work.

**Alternatives considered**

Adding provider calls to task reads would delay task loading.
Company-wide reuse of personal GitHub results would cross viewer access
boundaries. A periodic server worker would create work when no page
needs the result.

**Roadmap alignment**

This extends the shipped rich-link feature. It does not duplicate a
roadmap project or add a new provider integration.

## What Changed

- Add an additive, repeat-safe `link_previews` migration. Store channel,
resource kind, free-form status, JSON metadata and freshness. The table
has no task relationship or GitHub status enum.
- Bound each company/viewer cache to 1,000 rows across concurrent server
instances. Remove rows expired for over 30 days during writes. Limit
provider lookups to 600 requests per minute per scope/process. Keep
cached batch reads available during backoff.
- Add a database store owned by the link service. Reuse canonical
results across tasks. Keep company and viewer isolation. Save hover and
idle results. Preserve last known status during an outage and cache
unavailable results.
- Add a board-only batch GET endpoint, including viewer-role access. It
reads at most 50 URLs with one indexed select and makes no provider
calls. Both endpoints use private, no-store HTTP responses.
- Add a page coordinator for mounted rich links. Reuse browser data,
share pending reads, and defer provider work until document load, core
request settlement and browser idle time. Limit automatic work to 200
unique links and three requests in flight. Remove queued work when links
leave the page. Keep encoded batch URLs below 6 KB.
- Keep exact navigation URLs, status styling, anchored cards and
keyboard access. Add a page-load Storybook case and browser regression.

The link service owns cache policy and provider calls. The browser
coordinator owns page timing. The shared contract adds only the batch
limit and response shape. The database adds storage because process and
browser caches cannot serve a new browser session. Plugin resolver
registration is outside this change; storage accepts future provider
channels and status values.

## Verification

- **All 93 affected tests pass:** 40 link service, real PostgreSQL and
HTTP cases, plus 53 OpenAPI cases. Two SQL regressions and one HTTP case
failed before the storage repair. Concurrent writers now retain at most
1,000 rows per company/viewer; cleanup removes long-expired rows and
keeps recent stale status. Request backoff preserves cached reads and
isolates companies/viewers. The earlier eight concurrent hydration
regressions also pass. Focused store/service/unit types pass. Commands:
`pnpm exec vitest run server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1` and
`pnpm exec vitest run server/src/__tests__/openapi-routes.test.ts
--maxWorkers=1`.
- **All 50 affected server tests passed before the later server-only
repairs** on the master-integrated source: 20 recovery/deadline cases,
one authenticated review-counts HTTP case, 10 link-service cases, 13
real PostgreSQL cache cases and six link-route cases. Command: `pnpm
exec vitest run server/src/__tests__/issue-recovery-deadlines.test.ts
server/src/__tests__/tool-connection-review-counts-http.test.ts
server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1`.
- Reproduced the CI watchdog assertion locally twice before the test
repair. Ordinary assignment recovery can win `wakeReason` while the
watchdog wake coalesces into the same successful run with its full
mandate. The test now verifies the persisted watchdog identity, watched
issue and stopped fingerprint, plus exactly one review run. All 21
recovery/HTTP cases passed before integrating master; all 50 affected
cases pass after integration. No runtime behavior was changed for this
repair.
- **Fresh static Storybook build and all 22 browser cases pass**, with
two workers and retries disabled. They cover states, anchored hover,
keyboard access, exact navigation, duplicate requests, reload
persistence, expiry and saved-status hydration followed by one idle
refresh. The first browser attempt could not launch because this session
lacked Chromium; installing the expected browser resolved setup. That
attempt had no behavioral verdict.
- The cache repair remains covered by four PostgreSQL red/green
regressions for expired memory and newer durable status during provider
failure/timeout, plus check-time precedence and failed-read recovery.
Earlier UI/database/focused server type and migration
safety/compatibility/replay checks remain unchanged-input evidence.
- Master `77ddf260bf9a3de3dcb8456a60f4f809d67401fd` is integrated
cleanly. Link migration 0489 and its snapshot chain are unchanged. SDK
source validation checks 545 inputs with no stale source identities or
changed runtime/lock pins. Whitespace validation passes.
- `pnpm verify --explain` returns exit 2 and requires broad CI. No full
local suite or deployed acceptance is claimed. Earlier local full-server
type checks exceeded memory; current-head CI supplies broad
type/build/test coverage.
- [Previous CI
43900336895](https://github.com/paperclipai/paperclip/actions/runs/43900336895)
passed full types/build but failed on embedded PostgreSQL startup
(`ECONNREFUSED`), the watchdog assertion above, and unrelated plan-task
browser cleanup (activity-log foreign key on company deletion). One
failed-jobs retry was requested. Its browser shard 7 now passes without
code changes; the exact deletion-race cause remains unproven. The
watchdog assertion is repaired in this revision. The previous failures
remain recorded, not counted as an overall pass.
- [Prior CI
43905482848](https://github.com/paperclipai/paperclip/actions/runs/43905482848)
passed full types, build and all eight browser shards. Several test jobs
were cancelled; the run has no overall pass. Its concurrent update
finding is repaired. [Next CI
43909197530](https://github.com/paperclipai/paperclip/actions/runs/43909197530)
also passed types, build and all browser shards, but several server jobs
were cancelled. Greptile and Paperclip Review gave that revision 5/5.
Its security scan found unbounded retained cache rows. The quota,
retention and request-limit repair above addresses that finding. All ten
review threads are resolved.
- [Current CI
43912947278](https://github.com/paperclipai/paperclip/actions/runs/43912947278),
fresh code reviews and a fresh security scan are pending on
`335d89ef1668dcdeb66bb2059512a7c2da29e273`. Auto-merge is disabled until
the current revision clears its gates.

Published head: `335d89ef1668dcdeb66bb2059512a7c2da29e273`. Local head:
`b37b87c704a2ba8a65233c689a6a743f4011ce14`. Both have tree
`643ebd19b9729b1e6a6bb695442269c3db32be3a`; GitHub tree creation
verified equality.

## Risks

- Each company/viewer retains at most 1,000 rows. Writes remove excess
rows and entries whose freshness expired over 30 days ago. Inactive
scopes remain bounded and are cleaned on the next write. Company
deletion removes all rows. No periodic worker runs. Scope writes use a
transaction lock; the existing deadline and concurrency budget cover
that work.
- Last known status can be stale until idle or hover work completes.
Provider failures keep that status through the negative-cache window.
- Automatic enrichment adds bounded provider traffic. It runs only for
mounted supported links, once per mount, with the existing timeout and
server concurrency limit.
- Cache storage is viewer-scoped because provider results can be
private. New viewers load their own results. This prevents reuse across
users with different repository access.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools, code
execution and browser automation. The runtime does not expose the exact
serving model ID or context-window size. No subagents were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
This commit is contained in:
authored and GitHub committed 2026-11-16 14:33:39 -06:00
1 parent 0e8447fe77
commit f33c4cf6f7
27 files changed
+1572 -65

No files matched your search

+65
View File
@@ -0,0 +1,65 @@
# Rich link status caching
Rich link metadata belongs to the link preview service. GitHub pull requests are
the first provider. Links retain their exact navigation URL and use a canonical
resource URL for cache identity.
`link_previews` stores company, viewer, channel, resource kind, canonical URL,
free-form status, JSON metadata, check time and freshness expiry. It has no task
foreign key. The same resource on different tasks reuses a record. Channel,
resource kind, status and metadata are not GitHub enums; future provider adapters
can use their own values. This change does not register plugin resolvers.
Personal provider access remains viewer-scoped. The unique index is
`(company_id, viewer_id, canonical_url)`. The empty viewer is reserved for local
trusted mode. Company/board authorization runs before cache reads. Neither cache
endpoint is accessible to agents. Personal results never enter `external_objects`.
A cache fill is derivative read data; it does not mutate tasks or produce activity
feed events. Cache HTTP responses use `private, no-store`.
The browser registers mounted rich links after render. Up to 200 unique links
per active company view are eligible for automatic enrichment; further links
still work on hover/focus. One request reads up to 50 cached previews using one
indexed database select, without provider calls or task joins. The read-only GET
route also supports viewer-role members. Encoded request URLs stay below 6 KB. Duplicate links
share one browser query. Fresh browser results take precedence over older database
reads. Cache failures are quiet and do not block the task page.
After document load, core query settlement and a 500ms quiet delay, browser idle
callbacks resolve stale or missing status with at most three requests in flight.
Hover/focus uses the same query and server path, so overlapping requests coalesce.
Queued links are removed on unmount and scope changes dispose the coordinator.
There is no recurring refresh, window-focus lookup or server background worker.
New links from subsequent page content get their own one-shot enrichment.
The existing provider timeout, fixed-host lookup, managed connection selection,
public-access checks, negative cache window and process concurrency bound remain.
Provider results are stored before the successful lookup response completes when
the store is available within the shared five-second deadline. Stalled SQL keeps
its concurrency slot until it settles; the deadline still releases the response.
Expired process entries re-read durable storage before provider work. Both batch
reads and lookup fallback use check time to retain the newest known status; a
longer cache lifetime does not make an older status newer. When the durable read
fails too, provider-failure backoff stays local, so it cannot overwrite a saved
status that this process could not read. Database outages fall back to the
optional process cache/provider path. Successful
and unavailable results normally remain fresh for one minute; provider backoff
can extend that window up to one hour. Expired records retain last known status
for immediate display. An outage keeps that status and saves the negative
freshness window, so the page does not repeatedly retry.
Each company/viewer scope retains at most 1,000 rows. A write removes rows beyond
that limit, keeping the newest check times, and rows whose freshness expired more
than 30 days before the write's check time. A transaction-scoped advisory lock
serializes writes for that scope across replicas, so concurrent URL misses cannot
exceed the cap. Cleanup makes no queries during page hydration and starts no
background worker. Inactive scopes remain bounded and are cleaned on their next
write; company deletion removes all their rows. Normal stale status remains
available until it ages out or the scope reaches its quota. Negative results use
the same cap and retention policy as known status.
The single-link lookup route also limits each company/viewer to 600 requests per
minute per server process, using the existing in-memory request limiter. Excess
requests return 429 with Retry-After. Cached batch reads remain available during
that backoff and do not allocate stored rows. The durable quota is enforced across
replicas and remains effective when process-local request limits reset.
@@ -0,0 +1,14 @@
CREATE TABLE IF NOT EXISTS "link_previews" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"company_id" uuid NOT NULL CONSTRAINT "link_previews_company_id_companies_id_fk" REFERENCES "public"."companies"("id") ON DELETE cascade,
"viewer_id" text NOT NULL,
"channel" text NOT NULL,
"resource_kind" text NOT NULL,
"canonical_url" text NOT NULL,
"status" text,
"metadata" jsonb DEFAULT '{}'::jsonb NOT NULL,
"checked_at" timestamp with time zone NOT NULL,
"expires_at" timestamp with time zone NOT NULL
);
--> statement-breakpoint
CREATE UNIQUE INDEX IF NOT EXISTS "link_previews_scope_url_uq" ON "link_previews" USING btree ("company_id","viewer_id","canonical_url");
@@ -1,6 +1,6 @@
{
"id": "9389124c-379a-4b64-9c20-c7b37538e7c8",
"prevId": "ceaa38c4-681b-430e-9578-0854650d85b3",
"id": "dc61c7f3-cf3d-44c5-9296-86973e5f0925",
"prevId": "8f935a52-5f1e-4afe-8d13-c1770ca33968",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -6582,6 +6582,111 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.automatic_run_brakes": {
"name": "automatic_run_brakes",
"schema": "",
"columns": {
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'closed'"
},
"reason": {
"name": "reason",
"type": "jsonb",
"primaryKey": false,
"notNull": true,
"default": "'{}'::jsonb"
},
"state_changed_at": {
"name": "state_changed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"opened_at": {
"name": "opened_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"next_probe_at": {
"name": "next_probe_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"probe_run_id": {
"name": "probe_run_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"probe_successes": {
"name": "probe_successes",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"release_window_started_at": {
"name": "release_window_started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"release_window_starts": {
"name": "release_window_starts",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"next_creation_at": {
"name": "next_creation_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {},
"foreignKeys": {
"automatic_run_brakes_company_id_companies_id_fk": {
"name": "automatic_run_brakes_company_id_companies_id_fk",
"tableFrom": "automatic_run_brakes",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.board_api_keys": {
"name": "board_api_keys",
"schema": "",
@@ -36029,6 +36134,78 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_company_finished_idx": {
"name": "heartbeat_runs_company_finished_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "finished_at",
"isExpression": false,
"asc": false,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"finished_at\" is not null",
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_settled_process_identity_idx": {
"name": "heartbeat_runs_settled_process_identity_idx",
"columns": [
{
"expression": "process_host_hash",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' and \"heartbeat_runs\".\"status\" in ('failed', 'cancelled', 'timed_out', 'interrupted')\n and \"heartbeat_runs\".\"finished_at\" is not null\n and (\"heartbeat_runs\".\"process_pid\" is not null or \"heartbeat_runs\".\"process_group_id\" is not null or \"heartbeat_runs\".\"process_started_at\" is not null)",
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_pending_api_stop_idx": {
"name": "heartbeat_runs_pending_api_stop_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"status\" = 'cancelled' and \"heartbeat_runs\".\"driver_kind\" = 'cursor_cloud_api'\n and (\"heartbeat_runs\".\"result_json\"->'executionCancellation'->>'state') = 'requested'",
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_company_created_at_desc_idx": {
"name": "heartbeat_runs_company_created_at_desc_idx",
"columns": [
@@ -40722,6 +40899,131 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.issue_recovery_deferrals": {
"name": "issue_recovery_deferrals",
"schema": "",
"columns": {
"queue": {
"name": "queue",
"type": "text",
"primaryKey": false,
"notNull": true
},
"issue_id": {
"name": "issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"due_at": {
"name": "due_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
}
},
"indexes": {},
"foreignKeys": {
"issue_recovery_deferrals_issue_id_issues_id_fk": {
"name": "issue_recovery_deferrals_issue_id_issues_id_fk",
"tableFrom": "issue_recovery_deferrals",
"tableTo": "issues",
"columnsFrom": [
"issue_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"issue_recovery_deferrals_company_id_companies_id_fk": {
"name": "issue_recovery_deferrals_company_id_companies_id_fk",
"tableFrom": "issue_recovery_deferrals",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {
"issue_recovery_deferrals_pk": {
"name": "issue_recovery_deferrals_pk",
"columns": [
"queue",
"issue_id"
]
}
},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.issue_recovery_full_passes": {
"name": "issue_recovery_full_passes",
"schema": "",
"columns": {
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"queue": {
"name": "queue",
"type": "text",
"primaryKey": false,
"notNull": true
},
"started_at": {
"name": "started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
}
},
"indexes": {},
"foreignKeys": {
"issue_recovery_full_passes_company_id_companies_id_fk": {
"name": "issue_recovery_full_passes_company_id_companies_id_fk",
"tableFrom": "issue_recovery_full_passes",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {
"issue_recovery_full_passes_pk": {
"name": "issue_recovery_full_passes_pk",
"columns": [
"company_id",
"queue"
]
}
},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.issue_reference_mentions": {
"name": "issue_reference_mentions",
"schema": "",
@@ -44779,6 +45081,123 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.link_previews": {
"name": "link_previews",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"viewer_id": {
"name": "viewer_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"channel": {
"name": "channel",
"type": "text",
"primaryKey": false,
"notNull": true
},
"resource_kind": {
"name": "resource_kind",
"type": "text",
"primaryKey": false,
"notNull": true
},
"canonical_url": {
"name": "canonical_url",
"type": "text",
"primaryKey": false,
"notNull": true
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": false
},
"metadata": {
"name": "metadata",
"type": "jsonb",
"primaryKey": false,
"notNull": true,
"default": "'{}'::jsonb"
},
"checked_at": {
"name": "checked_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"expires_at": {
"name": "expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
}
},
"indexes": {
"link_previews_scope_url_uq": {
"name": "link_previews_scope_url_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "viewer_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "canonical_url",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"link_previews_company_id_companies_id_fk": {
"name": "link_previews_company_id_companies_id_fk",
"tableFrom": "link_previews",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.managed_agent_profiles": {
"name": "managed_agent_profiles",
"schema": "",
+7
View File
@@ -3410,6 +3410,13 @@
"when": 1794820062730,
"tag": "0488_issue_recovery_full_passes",
"breakpoints": true
},
{
"idx": 489,
"version": "7",
"when": 1794841384470,
"tag": "0489_real_sage",
"breakpoints": true
}
]
}
+1
View File
@@ -290,3 +290,4 @@ export { projectTaskExecutionPolicies, executionAdmissions, executionAdmissionPr
export { budgetEvaluationWork } from "./budget_evaluation_work.js";
export { tailscaleConnections } from "./tailscale_connections.js";
export { nativeProviderArtifacts, nativeProviderArtifactReferences } from "./native_provider_artifacts.js";
export { linkPreviews } from "./link_previews.js";
+19
View File
@@ -0,0 +1,19 @@
import { jsonb, pgTable, text, timestamp, uniqueIndex, uuid } from "drizzle-orm/pg-core";
import { companies } from "./companies.js";
/** Optional enrichment, independent of tasks and of provider-specific status vocabularies. */
export const linkPreviews = pgTable("link_previews", {
id: uuid("id").primaryKey().defaultRandom(),
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
// Personal provider results must not be reused by another viewer. Empty is local-trusted.
viewerId: text("viewer_id").notNull(),
channel: text("channel").notNull(),
resourceKind: text("resource_kind").notNull(),
canonicalUrl: text("canonical_url").notNull(),
status: text("status"),
metadata: jsonb("metadata").$type<Record<string, unknown>>().notNull().default({}),
checkedAt: timestamp("checked_at", { withTimezone: true }).notNull(),
expiresAt: timestamp("expires_at", { withTimezone: true }).notNull(),
}, table => ({
identity: uniqueIndex("link_previews_scope_url_uq").on(table.companyId, table.viewerId, table.canonicalUrl),
}));
+1 -1
View File
@@ -2976,7 +2976,7 @@ export {
TASK_IDENTITY_CHANGED_FIELD,
issueActivityChangesTaskIdentity,
} from "./issue-activity-identity.js";
export { recognizeLinkPreview, type LinkPreviewTarget, type LinkPreviewState, type LinkPreview, type LinkPreviewResult } from "./link-preview.js";
export { LINK_PREVIEW_BATCH_LIMIT, type LinkPreviewBatchResult, recognizeLinkPreview, type LinkPreviewTarget, type LinkPreviewState, type LinkPreview, type LinkPreviewResult } from "./link-preview.js";
export * from "./mcp-apps.js";
export * from "./tool-call-progress.js";
export * from "./mcp-skills.js";
+4 -1
View File
@@ -18,10 +18,13 @@ export interface LinkPreview {
export interface LinkPreviewResult {
preview: LinkPreview | null;
/** Demand-driven cache lifetime; this does not schedule a refresh. */
/** Remaining cache lifetime; zero means stale. This does not schedule a refresh. */
maxAgeMs: number;
}
export const LINK_PREVIEW_BATCH_LIMIT = 50;
export interface LinkPreviewBatchResult { results: Record<string, LinkPreviewResult> }
/** Recognition only. Never contacts a remote service or changes navigation. */
export function recognizeLinkPreview(href: string | null | undefined): LinkPreviewTarget | null {
if (!href || href.length > 2048) return null;
@@ -441,10 +441,19 @@ describeIssueRecovery("issue recovery production producers and deadlines", () =>
const [row] = await state.db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, watchdog.watchdog.id));
expect(row.watchdogIssueId).toBeTruthy(); reviewId = row.watchdogIssueId!;
}, { timeout: 15_000 });
await completed(reviewId!);
const review = await completed(reviewId!);
expect(failures).toEqual([]);
// Ordinary assignment recovery can win the wake reason while the watchdog
// wake coalesces into that run. Verify its mandate and one dispatch instead.
expect(review.contextSnapshot).toMatchObject({
watchdogId: watchdog.watchdog.id,
taskWatchdog: {
watchedIssueId: source.id,
stopFingerprint: expect.stringMatching(/^task_watchdog_stop:/),
},
});
expect(await state.db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'task_watchdog_stopped_subtree'`))).toHaveLength(1);
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${reviewId}`))).toHaveLength(1);
await finish(reviewId!);
await taskWatchdogService(state.db).disableForIssue(f.companyId, source.id);
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
@@ -4,21 +4,44 @@ import { describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
import { linkPreviewRoutes } from "../routes/link-previews.js";
import { HttpError } from "../errors.js";
import { createInviteRateLimiter, type InviteRateLimiter } from "../services/invite-rate-limit.js";
function app(actor: express.Request["actor"]) {
function app(actor: express.Request["actor"], limiter?: InviteRateLimiter) {
const get = vi.fn(async () => ({ preview: null, maxAgeMs: 60_000 }));
const cached = vi.fn(async () => ({ results: {} }));
const server = express();
server.use(express.json());
server.use((req, _res, next) => { req.actor = actor; next(); });
server.use(linkPreviewRoutes(null as unknown as Db, { get }));
server.use(linkPreviewRoutes(null as unknown as Db, { get, cached }, limiter));
server.use((error: HttpError, _req: express.Request, res: express.Response, _next: express.NextFunction) => {
res.status(error.status ?? 500).json({ error: error.message });
});
return { server, get };
return { server, get, cached };
}
const board = { type: "board", source: "session", userId: "alice", companyIds: ["company-a"] } as const;
const path = "/companies/company-a/link-preview";
describe("link preview routes", () => {
it("limits lookups per viewer and company without blocking cached reads", async () => {
let now = 0;
const limiter = createInviteRateLimiter({ maxRequests: 2, windowMs: 1_000, now: () => now });
const actor = { ...board, userId: "alice", companyIds: ["company-a", "company-b"] };
const { server, get, cached } = app(actor, limiter);
const query = { url: "https://github.com/acme/web/pull/42" };
for (let n = 0; n < 2; n++) expect((await request(server).get(path).query(query)).status).toBe(200);
const limited = await request(server).get(path).query(query);
expect(limited.status).toBe(429);
expect(limited.headers["retry-after"]).toBe("1");
expect(get).toHaveBeenCalledTimes(2);
expect((await request(server).get("/companies/company-a/link-previews/cached").query(query)).status).toBe(200);
expect(cached).toHaveBeenCalledTimes(1);
expect((await request(server).get("/companies/company-b/link-preview").query(query)).status).toBe(200);
actor.userId = "bob";
expect((await request(server).get(path).query(query)).status).toBe(200);
actor.userId = "alice";
now = 1_001;
expect((await request(server).get(path).query(query)).status).toBe(200);
});
it("passes the authenticated viewer and disables shared HTTP caching", async () => {
const { server, get } = app({ ...board, companyIds: [...board.companyIds] });
const response = await request(server).get(path).query({ url: "https://github.com/acme/web/pull/42" });
@@ -38,6 +61,25 @@ describe("link preview routes", () => {
expect(get).not.toHaveBeenCalled();
}
});
it("reads a bounded batch only after viewer and company authorization", async () => {
const { server, cached, get } = app({ ...board, companyIds: [...board.companyIds] });
const response = await request(server).get("/companies/company-a/link-previews/cached").query({ url: ["https://github.com/acme/web/pull/42"] });
expect(response.status).toBe(200);
expect(response.headers["cache-control"]).toBe("private, no-store");
expect(cached).toHaveBeenCalledWith({ companyId: "company-a", userId: "alice" }, ["https://github.com/acme/web/pull/42"]);
expect(get).not.toHaveBeenCalled();
for (const body of [{}, { url: ["x".repeat(2049)] }, { url: Array(51).fill("a") }]) {
expect((await request(server).get("/companies/company-a/link-previews/cached").query(body)).status).toBe(400);
}
const denied = app({ ...board, companyIds: ["company-b"] });
expect((await request(denied.server).get("/companies/company-a/link-previews/cached")).status).toBe(403);
expect(denied.cached).not.toHaveBeenCalled();
});
it("lets active viewer-role members read stored statuses", async () => {
const { server, cached } = app({ ...board, companyIds: [...board.companyIds], memberships: [{ companyId: "company-a", membershipRole: "viewer", status: "active" }] });
expect((await request(server).get("/companies/company-a/link-previews/cached").query({ url: "https://github.com/acme/web/pull/42" })).status).toBe(200);
expect(cached).toHaveBeenCalledTimes(1);
});
it("bounds and validates the URL input", async () => {
const { server, get } = app({ ...board, companyIds: [...board.companyIds] });
for (const query of [{}, { url: "x".repeat(2049) }, { url: ["a", "b"] }]) {
@@ -0,0 +1,253 @@
import { randomUUID } from "node:crypto";
import { readFile } from "node:fs/promises";
import { companies, linkPreviews } from "@paperclipai/db";
import { recognizeLinkPreview } from "@paperclipai/shared";
import { sql } from "drizzle-orm";
import { expect, it, vi } from "vitest";
import { describeEmbeddedPostgres, useEmbeddedPostgres } from "./helpers/route-test-harness.js";
import { createLinkPreviewStore } from "../services/link-preview-store.js";
import { createLinkPreviewService } from "../services/link-previews.js";
const url = "https://github.com/acme/web/pull/42";
const target = recognizeLinkPreview(url)!;
const preview = { target, title: "Durable PR", state: "merged" as const, author: "alice", updatedAt: null };
describeEmbeddedPostgres("durable link preview cache", () => {
const context = useEmbeddedPostgres("link-previews-");
async function fixture() {
const companyId = randomUUID();
await context.db.insert(companies).values({ id: companyId, name: "Link cache", issuePrefix: companyId });
return { companyId, userId: "alice" };
}
it("bounds retained rows across concurrent store instances and isolates their quotas", async () => {
const scope = await fixture();
const other = { ...scope, userId: "bob" };
const now = Date.now();
const seed = Array.from({ length: 1_000 }, (_, n) => ({
companyId: scope.companyId, viewerId: scope.userId, channel: "github", resourceKind: "pull_request",
canonicalUrl: url + n, status: null, metadata: {}, checkedAt: new Date(now - 1_000 + n), expiresAt: new Date(now + 60_000),
}));
await context.db.insert(linkPreviews).values([...seed, { ...seed[0], viewerId: other.userId }]);
const first = createLinkPreviewStore(context.db);
const second = createLinkPreviewStore(context.db);
await Promise.all(Array.from({ length: 8 }, (_, n) => (n % 2 ? first : second).write(scope,
recognizeLinkPreview(url + (1_000 + n))!, { result: { preview: null, maxAgeMs: 60_000 }, expiresAt: now + 60_000 }, now + n)));
const rows = await context.db.select().from(linkPreviews).where(sql`${linkPreviews.companyId} = ${scope.companyId}`);
expect(rows.filter(row => row.viewerId === scope.userId)).toHaveLength(1_000);
expect(rows.filter(row => row.viewerId === other.userId)).toHaveLength(1);
const saved = await first.read(scope, [recognizeLinkPreview(url + "0")!, recognizeLinkPreview(url + "1007")!]);
expect(saved.has(url + "0")).toBe(false);
expect(saved.has(url + "1007")).toBe(true);
});
it("removes long-expired rows on writes and preserves recent stale status", async () => {
const scope = await fixture();
const now = Date.now();
const expired = now - 31 * 24 * 60 * 60_000;
await context.db.insert(linkPreviews).values([
{ companyId: scope.companyId, viewerId: scope.userId, channel: "github", resourceKind: "pull_request", canonicalUrl: url,
status: "merged", metadata: { title: preview.title }, checkedAt: new Date(expired - 60_000), expiresAt: new Date(expired) },
{ companyId: scope.companyId, viewerId: scope.userId, channel: "github", resourceKind: "pull_request", canonicalUrl: url + "1",
status: "merged", metadata: { title: preview.title }, checkedAt: new Date(now - 120_000), expiresAt: new Date(now - 60_000) },
]);
const store = createLinkPreviewStore(context.db);
await store.write(scope, recognizeLinkPreview(url + "2")!, { result: { preview: null, maxAgeMs: 60_000 }, expiresAt: now + 60_000 }, now);
const saved = await store.read(scope, [target, recognizeLinkPreview(url + "1")!, recognizeLinkPreview(url + "2")!]);
expect(saved.has(url)).toBe(false);
expect(saved.get(url + "1")?.result.preview?.state).toBe("merged");
expect(saved.has(url + "2")).toBe(true);
});
it.each(["failure", "success", "timeout", "stale timeout"])("keeps another replica's concurrent hydration after provider %s", async (settlement) => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
const oldPreview = { ...preview, state: "open" as const };
await store.write(scope, target, { result: { preview: oldPreview, maxAgeMs: 1_000 }, expiresAt: 2_000 }, now);
let finish!: (value: { preview: typeof oldPreview | null; maxAgeMs: number }) => void;
let started!: () => void;
const providerStarted = new Promise<void>(done => { started = done; });
const resolve = vi.fn(() => new Promise<{ preview: typeof oldPreview | null; maxAgeMs: number }>(done => { finish = done; started(); }));
const write = vi.fn(store.write);
const first = createLinkPreviewService([{ key: "github", resolve }], { store: { ...store, write }, now: () => now, timeoutMs: 1_000 });
await first.cached(scope, [url]);
now = 3_000;
const pending = first.get(scope, url);
await providerStarted;
now = 4_000;
const expiresAt = settlement === "stale timeout" ? 3_500 : 34_000;
await store.write(scope, target, { result: { preview, maxAgeMs: 30_000 }, expiresAt }, now);
expect((await first.cached(scope, [url])).results[url].preview).toEqual(preview);
if (!settlement.includes("timeout")) finish({ preview: settlement === "success" ? oldPreview : null, maxAgeMs: 60_000 });
expect(await pending).toEqual({ preview, maxAgeMs: Math.max(0, expiresAt - now) });
// A second batch checks that the process did not retain the older status.
expect((await first.cached(scope, [url])).results[url].preview).toEqual(preview);
const saved = (await store.read(scope, [target])).get(url)!;
expect(saved.result.preview).toEqual(preview);
expect(saved.checkedAt).toBe(4_000);
expect(saved.expiresAt).toBe(expiresAt);
expect(write).not.toHaveBeenCalled();
finish({ preview: null, maxAgeMs: 60_000 });
});
it.each(["open", "negative"])("rechecks expired %s memory before calling the provider", async (initialState) => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
const resolve = vi.fn(async () => ({ preview: initialState === "open" ? { ...preview, state: "open" as const } : null, maxAgeMs: 60_000 }));
const first = createLinkPreviewService([{ key: "github", resolve }], { store, now: () => now });
await first.get(scope, url);
now = 62_000;
const other = createLinkPreviewService([{ key: "github", resolve: async () => ({ preview, maxAgeMs: 60_000 }) }], { store, now: () => now });
await other.get(scope, url);
resolve.mockRejectedValue(new Error("provider unavailable"));
expect((await first.get(scope, url)).preview).toEqual(preview);
expect(resolve).toHaveBeenCalledTimes(1);
const saved = (await store.read(scope, [target])).get(url)!;
expect(saved.result.preview).toEqual(preview);
expect(saved.expiresAt).toBe(122_000);
});
it.each(["failure", "timeout"])("retains the newest stale stored preview on provider %s", async (failure) => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
let requests = 0;
const resolve = async (_target: import("@paperclipai/shared").LinkPreviewTarget, _scope: import("../services/link-previews.js").LinkPreviewScope, signal: AbortSignal) => {
if (++requests === 1) return { preview: { ...preview, state: "open" as const }, maxAgeMs: 60_000 };
if (failure === "failure") throw new Error("provider unavailable");
return new Promise<never>((_done, reject) => signal.addEventListener("abort", () => reject(new Error("timeout"))));
};
const first = createLinkPreviewService([{ key: "github", resolve }], { store, now: () => now, timeoutMs: 100 });
await first.get(scope, url);
now = 62_000;
await store.write(scope, target, { result: { preview, maxAgeMs: 1_000 }, expiresAt: 63_000 }, now);
now = 64_000;
expect((await first.get(scope, url)).preview).toEqual(preview);
await vi.waitFor(async () => expect((await store.read(scope, [target])).get(url)?.expiresAt).toBe(124_000));
expect((await store.read(scope, [target])).get(url)?.result.preview).toEqual(preview);
});
it("compares check times rather than cache lifetimes when hydrating", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
await store.write(scope, target, { result: { preview: { ...preview, state: "open" }, maxAgeMs: 60_000 }, expiresAt: 61_000 }, now);
const service = createLinkPreviewService([{ key: "github", resolve: async () => ({ preview: null, maxAgeMs: 60_000 }) }], { store, now: () => now });
await service.cached(scope, [url]);
await store.write(scope, target, { result: { preview, maxAgeMs: 1_000 }, expiresAt: 4_000 }, 3_000);
now = 5_000;
expect((await service.cached(scope, [url])).results[url]).toEqual({ preview, maxAgeMs: 0 });
});
it("keeps a failed durable read's fallback local and accepts recovered stored status", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const read = vi.fn(store.read);
let now = 1_000;
const resolve = vi.fn(async () => ({ preview: { ...preview, state: "open" as const }, maxAgeMs: 60_000 }));
const first = createLinkPreviewService([{ key: "github", resolve }], { store: { ...store, read }, now: () => now });
await first.get(scope, url);
now = 62_000;
await store.write(scope, target, { result: { preview, maxAgeMs: 60_000 }, expiresAt: 122_000 }, now);
now = 64_000;
read.mockRejectedValueOnce(new Error("temporary database outage"));
resolve.mockRejectedValueOnce(new Error("provider unavailable"));
expect((await first.get(scope, url)).preview?.state).toBe("open");
const saved = (await store.read(scope, [target])).get(url)!;
expect(saved.result.preview).toEqual(preview);
expect(saved.expiresAt).toBe(122_000);
expect((await first.cached(scope, [url])).results[url].preview).toEqual(preview);
});
it.each(["missing", "failed"])("reads another replica's saved status after a %s batch read", async (initialRead) => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const read = vi.fn(store.read);
if (initialRead === "failed") read.mockRejectedValueOnce(new Error("temporary database outage"));
const write = vi.fn(store.write);
const resolve = vi.fn(async () => { throw new Error("provider unavailable"); });
const first = createLinkPreviewService([{ key: "github", resolve }], { store: { read, write } });
expect(await first.cached(scope, [url])).toEqual({ results: {} });
const other = createLinkPreviewService([{ key: "github", resolve: async () => ({ preview, maxAgeMs: 60_000 }) }], { store });
await other.get(scope, url);
expect((await first.get(scope, url)).preview).toEqual(preview);
expect(read).toHaveBeenCalledTimes(2);
expect(resolve).not.toHaveBeenCalled();
expect(write).not.toHaveBeenCalled();
expect((await store.read(scope, [target])).get(url)?.result.preview).toEqual(preview);
});
it("survives service recreation, reuses canonical URLs across tasks, and isolates viewers/companies", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const resolve = vi.fn(async () => ({ preview, maxAgeMs: 60_000 }));
const make = () => createLinkPreviewService([{ key: "github", resolve }], { store });
await make().get(scope, url);
const restored = await make().cached(scope, [url, "https://www.github.com/ACME/WEB/pull/42/files#review"]);
expect(Object.keys(restored.results)).toEqual([url]);
expect(restored.results[url].preview).toMatchObject({ title: preview.title, state: "merged", target: { canonicalUrl: url } });
expect(restored.results[url].maxAgeMs).toBeGreaterThan(0);
await make().get(scope, url);
expect(resolve).toHaveBeenCalledTimes(1);
expect(await make().cached({ ...scope, userId: "bob" }, [url])).toEqual({ results: {} });
expect(await make().cached(await fixture(), [url])).toEqual({ results: {} });
});
it("returns stale status immediately, refreshes on demand, and persists negative freshness", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
let now = 1_000;
const resolve = vi.fn(async () => ({ preview, maxAgeMs: 60_000 }));
const make = () => createLinkPreviewService([{ key: "github", resolve }], { store, now: () => now });
await make().get(scope, url);
now += 61_000;
expect((await make().cached(scope, [url])).results[url]).toEqual({ preview, maxAgeMs: 0 });
expect(resolve).toHaveBeenCalledTimes(1);
resolve.mockRejectedValueOnce(new Error("outage"));
expect((await make().get(scope, url)).preview).toEqual(preview);
await make().get(scope, url);
expect(resolve).toHaveBeenCalledTimes(2);
const missing = url + "1";
resolve.mockRejectedValueOnce(new Error("private"));
await make().get(scope, missing);
await make().get(scope, missing);
expect(resolve).toHaveBeenCalledTimes(3);
expect((await make().cached(scope, [missing])).results[missing].preview).toBeNull();
});
it("persists provider timeout freshness across service restarts without extending the response", async () => {
const scope = await fixture();
const database = createLinkPreviewStore(context.db);
let saved!: () => void;
const persisted = new Promise<void>(done => { saved = done; });
const store = { ...database, write: async (...args: Parameters<typeof database.write>) => { await database.write(...args); saved(); } };
const resolve = vi.fn((_target: import("@paperclipai/shared").LinkPreviewTarget, _scope: import("../services/link-previews.js").LinkPreviewScope, signal: AbortSignal) => new Promise<never>((_done, reject) => {
signal.addEventListener("abort", () => reject(new Error("timeout")));
}));
const first = createLinkPreviewService([{ key: "github", resolve }], { store, timeoutMs: 100 });
expect((await first.get(scope, url)).preview).toBeNull();
await persisted;
const restarted = createLinkPreviewService([{ key: "github", resolve }], { store });
expect((await restarted.get(scope, url)).preview).toBeNull();
expect(resolve).toHaveBeenCalledTimes(1);
expect((await restarted.cached(scope, [url])).results[url].maxAgeMs).toBeGreaterThan(0);
});
it("reads a 50-link page in one indexed select and does not require task relationships", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const targets = Array.from({ length: 50 }, (_, n) => recognizeLinkPreview(url + n)!);
await Promise.all(targets.map(t => store.write(scope, t, { result: { preview: { ...preview, target: t }, maxAgeMs: 60_000 }, expiresAt: 61_000 }, 1_000)));
const select = vi.spyOn(context.db, "select");
expect((await store.read(scope, targets)).size).toBe(50);
expect(select).toHaveBeenCalledTimes(1);
select.mockRestore();
const plan = await context.db.execute(sql`EXPLAIN SELECT * FROM link_previews WHERE company_id = ${scope.companyId} AND viewer_id = 'alice' AND canonical_url IN (${sql.join(targets.map(t => sql`${t.canonicalUrl}`), sql`, `)})`);
expect(JSON.stringify(plan)).toContain("link_previews_scope_url_uq");
});
it("supports other channels/statuses and older writers cannot replace newer results", async () => {
const scope = await fixture();
const store = createLinkPreviewStore(context.db);
const deployment = { ...target, provider: "vercel", kind: "deployment", canonicalUrl: "https://vercel.com/acme/build/123" };
await context.db.insert(linkPreviews).values({ companyId: scope.companyId, viewerId: scope.userId, channel: deployment.provider,
resourceKind: deployment.kind, canonicalUrl: deployment.canonicalUrl, status: "deployed", metadata: { deploymentId: "123", region: "iad1" }, checkedAt: new Date(), expiresAt: new Date() });
await store.write(scope, target, { result: { preview, maxAgeMs: 60_000 }, expiresAt: 65_000 }, 5_000);
await store.write(scope, target, { result: { preview: { ...preview, state: "open" }, maxAgeMs: 60_000 }, expiresAt: 61_000 }, 1_000);
expect((await store.read(scope, [target])).get(url)?.result.preview?.state).toBe("merged");
const migration = await readFile(new URL("../../../packages/db/src/migrations/0489_real_sage.sql", import.meta.url), "utf8");
for (const statement of migration.split("--> statement-breakpoint")) if (statement.trim()) await context.db.execute(sql.raw(statement));
expect((await store.read(scope, [target])).get(url)?.result.preview?.state).toBe("merged");
});
});
@@ -2,6 +2,8 @@ import { createServer } from "node:http";
import { once } from "node:events";
import { afterEach, describe, expect, it, vi } from "vitest";
import { createLinkPreviewService, type LinkPreviewProvider } from "../services/link-previews.js";
import type { StoredLinkPreview } from "../services/link-preview-store.js";
import { recognizeLinkPreview } from "@paperclipai/shared";
const scope = { companyId: "company-a", userId: "alice" };
const url = "https://github.com/acme/web/pull/42";
@@ -10,6 +12,51 @@ const provider = (resolve = vi.fn(async () => unavailable)): LinkPreviewProvider
afterEach(() => vi.useRealTimers());
describe("demand-driven link preview cache", () => {
it.each(["failure", "success", "timeout", "write"])("keeps concurrent hydration newer than a pending provider %s", async (settlement) => {
vi.useFakeTimers();
let now = 1_000;
const target = recognizeLinkPreview(url)!;
const oldPreview = { target, title: "Old status", state: "open" as const, author: null, updatedAt: null };
const newPreview = { ...oldPreview, title: "New status", state: "merged" as const };
let stored: StoredLinkPreview = { result: { preview: oldPreview, maxAgeMs: 1_000 }, expiresAt: 2_000, checkedAt: now };
let finish!: (value: typeof unavailable | { preview: typeof oldPreview; maxAgeMs: number }) => void;
let finishWrite!: () => void;
const write = vi.fn(async () => {
if (settlement === "write") await new Promise<void>(done => { finishWrite = done; });
});
const resolve = vi.fn(() => new Promise<typeof unavailable | { preview: typeof oldPreview; maxAgeMs: number }>(done => { finish = done; }));
const service = createLinkPreviewService([{ key: "github", resolve }], {
now: () => now, timeoutMs: 20,
store: { read: async () => new Map([[url, stored]]), write },
});
await service.cached(scope, [url]);
now = 3_000;
const pending = service.get(scope, url);
await vi.advanceTimersByTimeAsync(0);
if (settlement === "write") {
finish({ preview: oldPreview, maxAgeMs: 60_000 });
await vi.advanceTimersByTimeAsync(0);
expect(write).toHaveBeenCalledTimes(1);
}
now = 4_000;
stored = { result: { preview: newPreview, maxAgeMs: 30_000 }, expiresAt: 34_000, checkedAt: now };
expect((await service.cached(scope, [url])).results[url].preview).toEqual(newPreview);
if (settlement === "write") finishWrite();
else if (settlement === "timeout") await vi.advanceTimersByTimeAsync(21);
else finish(settlement === "failure" ? unavailable : { preview: oldPreview, maxAgeMs: 60_000 });
expect(await pending).toEqual({ preview: newPreview, maxAgeMs: 30_000 });
expect(await service.get(scope, url)).toEqual({ preview: newPreview, maxAgeMs: 30_000 });
expect(resolve).toHaveBeenCalledTimes(1);
expect(write).toHaveBeenCalledTimes(settlement === "write" ? 1 : 0);
if (settlement === "timeout") { finish(unavailable); await vi.advanceTimersByTimeAsync(0); }
});
it("omits absent records on repeated batch reads", async () => {
const service = createLinkPreviewService([provider()], {
store: { read: async () => new Map(), write: async () => {} },
});
expect(await service.cached(scope, [url])).toEqual({ results: {} });
expect(await service.cached(scope, [url])).toEqual({ results: {} });
});
it("does no work until requested and coalesces equivalent URLs", async () => {
const resolve = vi.fn(async () => unavailable);
const service = createLinkPreviewService([provider(resolve)]);
@@ -86,6 +133,36 @@ describe("demand-driven link preview cache", () => {
expect(resolve).toHaveBeenCalledTimes(2);
finish(unavailable);
});
it("bounds a stalled database write and holds its concurrency slot until settlement", async () => {
vi.useFakeTimers();
let finish!: () => void;
const write = vi.fn(async () => {}).mockImplementationOnce(() => new Promise<void>(done => { finish = done; }));
const resolve = vi.fn(async (target: import("@paperclipai/shared").LinkPreviewTarget) => ({ preview: { target, title: "Known", state: "open" as const, author: null, updatedAt: null }, maxAgeMs: 60_000 }));
const service = createLinkPreviewService([{ key: "github", resolve }], {
store: { read: async () => new Map(), write }, timeoutMs: 20, maxConcurrent: 1,
});
const first = service.get(scope, url);
await vi.advanceTimersByTimeAsync(21);
expect((await first).preview?.state).toBe("open");
expect(await service.get(scope, url + "1")).toEqual(unavailable);
expect(resolve).toHaveBeenCalledTimes(1);
finish(); await vi.advanceTimersByTimeAsync(1);
await service.get(scope, url + "1");
expect(resolve).toHaveBeenCalledTimes(2);
});
it("keeps timeout persistence inside the concurrency budget after the provider aborts", async () => {
vi.useFakeTimers();
let finishWrite!: () => void;
const write = vi.fn(async () => {}).mockImplementationOnce(() => new Promise<void>(done => { finishWrite = done; }));
const resolve = vi.fn(async (_target: import("@paperclipai/shared").LinkPreviewTarget, _scope: import("../services/link-previews.js").LinkPreviewScope, signal: AbortSignal) => unavailable)
.mockImplementationOnce((_target, _scope, signal) => new Promise((_done, reject) => signal.addEventListener("abort", () => reject(new Error("timeout")))));
const service = createLinkPreviewService([{ key: "github", resolve }], { store: { read: async () => new Map(), write }, timeoutMs: 20, maxConcurrent: 1 });
const first = service.get(scope, url);
await vi.advanceTimersByTimeAsync(21); expect(await first).toEqual(unavailable);
await service.get(scope, url + "1"); expect(resolve).toHaveBeenCalledTimes(1);
finishWrite(); await vi.advanceTimersByTimeAsync(1);
await service.get(scope, url + "1"); expect(resolve).toHaveBeenCalledTimes(2);
});
it("aborts a real stalled HTTP read while an unrelated lookup remains responsive", async () => {
const server = createServer((_req, res) => { res.writeHead(200, { "content-type": "application/json" }); res.write("{"); });
server.listen(0, "127.0.0.1");
+10 -1
View File
@@ -4,7 +4,7 @@ import path from "node:path";
import { fileURLToPath } from "node:url";
import request from "supertest";
import { describe, expect, it } from "vitest";
import { MUSE_ASSET_VERSION, MUSE_PUBLIC_ROUTES } from "@paperclipai/shared";
import { LINK_PREVIEW_BATCH_LIMIT, MUSE_ASSET_VERSION, MUSE_PUBLIC_ROUTES } from "@paperclipai/shared";
import { COMPANY_IMPORT_TRANSFERS_ROUTE_PATH } from "@paperclipai/shared/company-import-transfer";
import { errorHandler } from "../middleware/index.js";
import { buildOpenApiSpec, openApiRoutes, applyPluginSettingsVisibility, buildOpenApiDocument } from "../routes/openapi.js";
@@ -338,6 +338,15 @@ function loadSpecRoutes() {
}
describe("openapi routes", () => {
it("describes cached link results with the lookup shape and shared batch bound", () => {
const paths = buildOpenApiSpec().paths;
const batch = paths["/api/companies/{companyId}/link-previews/cached"].get;
expect(batch.description).toContain(`at most ${LINK_PREVIEW_BATCH_LIMIT} URLs`);
const lookupResult = paths["/api/companies/{companyId}/link-preview"].get.responses[200].content["application/json"].schema;
expect(batch.responses[200].content["application/json"].schema.properties.results.additionalProperties).toEqual(lookupResult);
expect(lookupResult.properties).toHaveProperty("preview");
expect(lookupResult.properties).toHaveProperty("maxAgeMs");
});
it("tells agents that unpriced usage in cost reports is unknown cost, not zero", () => {
const paths = buildOpenApiSpec().paths;
for (const segment of ["summary", "by-agent", "by-agent-model", "by-project"]) {
+24 -2
View File
@@ -1,11 +1,15 @@
import { Router } from "express";
import type { Db } from "@paperclipai/db";
import { badRequest } from "../errors.js";
import { LINK_PREVIEW_BATCH_LIMIT } from "@paperclipai/shared";
import { createLinkPreviewStore } from "../services/link-preview-store.js";
import { badRequest, tooManyRequests } from "../errors.js";
import { createInviteRateLimiter, type InviteRateLimiter } from "../services/invite-rate-limit.js";
import { assertBoard, assertCompanyAccess } from "./authz.js";
import { createLinkPreviewService } from "../services/link-previews.js";
import { createGitHubLinkPreviewProvider } from "../services/github-link-preview.js";
export function linkPreviewRoutes(db: Db, service = createLinkPreviewService([createGitHubLinkPreviewProvider(db)])) {
export function linkPreviewRoutes(db: Db, service = createLinkPreviewService([createGitHubLinkPreviewProvider(db)], { store: createLinkPreviewStore(db) }),
limiter: InviteRateLimiter = createInviteRateLimiter({ maxRequests: 600 })) {
const router = Router();
router.get("/companies/:companyId/link-preview", async (req, res) => {
const companyId = req.params.companyId as string;
@@ -15,7 +19,25 @@ export function linkPreviewRoutes(db: Db, service = createLinkPreviewService([cr
if (typeof href !== "string" || href.length > 2048) throw badRequest("A link URL is required");
// Personal repository metadata must never be stored by a proxy or shared browser cache.
res.set("Cache-Control", "private, no-store");
const limit = limiter.consume(JSON.stringify([companyId, req.actor.userId ?? null]));
res.set("X-RateLimit-Limit", String(limit.limit));
res.set("X-RateLimit-Remaining", String(limit.remaining));
if (!limit.allowed) {
res.set("Retry-After", String(limit.retryAfterSeconds));
throw tooManyRequests("Link preview rate limit exceeded");
}
res.json(await service.get({ companyId, userId: req.actor.userId ?? null }, href));
});
// Read-only batch: page hydration never waits for a remote provider.
router.get("/companies/:companyId/link-previews/cached", async (req, res) => {
const companyId = req.params.companyId as string;
assertCompanyAccess(req, companyId);
assertBoard(req);
const urls = typeof req.query.url === "string" ? [req.query.url] : req.query.url;
if (!Array.isArray(urls) || !urls.length || urls.length > LINK_PREVIEW_BATCH_LIMIT
|| urls.some(url => typeof url !== "string" || url.length > 2048)) throw badRequest("A bounded list of link URLs is required");
res.set("Cache-Control", "private, no-store");
res.json(await service.cached({ companyId, userId: req.actor.userId ?? null }, urls as string[]));
});
return router;
}
+26 -11
View File
@@ -2,7 +2,7 @@ import { catalogTeamInstallSchema, catalogTeamPreviewSchema, updateWorkspacePref
import { MUSE_ASSET_VERSION, museStopBoundarySchema, museDetectorCleanupAttestationSchema, createGrokInvitationSchema, configureGrokRoutineSchema, verifyGrokRoutineSchema, recoverGrokRoutineSchema, pairGrokBindingSchema, stopGrokWorkSchema, retryGrokDeliverySchema, updateFastResponseSchema } from "@paperclipai/shared";
import { slackTestMessageSchema, slackTestMessageResultSchema, slackManagerAuthorizationSchema, slackSetupOptionsSchema, slackManagedProvisionSchema, slackManagedResultSchema, slackRegistrationSchema, slackSetupActionSchema, slackInstallAuthorizationSchema, slackRegistrationStateSchema, slackAppConfigurationSchema, slackAvatarStateSchema, slackAccountStateSchema } from "@paperclipai/shared";
import { experimentalApiMetadata } from "./experimental-api-metadata.js";
import { ATTENTION_SOURCE_KINDS } from "@paperclipai/shared";
import { LINK_PREVIEW_BATCH_LIMIT, ATTENTION_SOURCE_KINDS } from "@paperclipai/shared";
import { agentCaseHandoffSchema, caseWorkflowDocumentTextSchema, doneCaseHandoffSchema, createWorkflowSchema, updateWorkflowSchema, listWorkflowsSchema, readWorkflowQuerySchema } from "@paperclipai/shared";
import {
experimentalApiDocs,
@@ -10459,26 +10459,41 @@ registry.registerPath({
responses: { 200: r.ok(), 401: r.unauthorized, 404: r.notFound },
});
const linkPreviewResultSchema = z.object({
preview: z.object({
target: z.object({ provider: z.string(), kind: z.string(), canonicalUrl: z.string(), label: z.string() }),
title: z.string(), state: z.enum(["unknown", "open", "draft", "merged", "closed", "conflicts"]),
author: z.string().nullable(), updatedAt: z.string().nullable(),
}).nullable(),
maxAgeMs: z.number(),
});
registry.registerPath({
method: "get",
path: "/api/companies/{companyId}/link-preview",
tags: ["companies"],
summary: "Read optional link metadata for the current board user",
description: "Company access and board authentication are required. This read uses a viewer-scoped cache and returns a quiet null preview when provider access is unavailable. It does not change tasks or schedule background work.",
description: "Company access and board authentication are required. This read uses a viewer-scoped cache and returns a quiet null preview when provider access is unavailable. Each company/viewer can make 600 lookup requests per minute per server process. Excess requests return 429 with Retry-After; cached batch reads remain available. It does not change tasks or schedule background work.",
request: { params: z.object({ companyId: z.string() }), query: z.object({ url: z.string().max(2048) }) },
responses: {
200: r.ok(z.object({
preview: z.object({
target: z.object({ provider: z.string(), kind: z.string(), canonicalUrl: z.string(), label: z.string() }),
title: z.string(), state: z.enum(["unknown", "open", "draft", "merged", "closed", "conflicts"]),
author: z.string().nullable(), updatedAt: z.string().nullable(),
}).nullable(),
maxAgeMs: z.number(),
})),
400: r.badRequest, 401: r.unauthorized, 403: r.forbidden,
200: r.ok(linkPreviewResultSchema),
400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 429: r.tooManyRequests,
},
});
registry.registerPath({
method: "get",
path: "/api/companies/{companyId}/link-previews/cached",
tags: ["companies"],
summary: "Read a batch of stored link statuses without contacting providers",
description: `Board-only and company/viewer scoped. Reads at most ${LINK_PREVIEW_BATCH_LIMIT} URLs. Canonical links are reused across tasks. Stale records have maxAgeMs zero; absent records are omitted. This is a read-only cache projection.`,
request: {
params: z.object({ companyId: z.string() }),
query: z.object({ url: z.union([z.string().max(2048), z.array(z.string().max(2048)).min(1).max(LINK_PREVIEW_BATCH_LIMIT)]) }),
},
responses: { 200: r.ok(z.object({ results: z.record(z.string(), linkPreviewResultSchema) })), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden },
});
registry.registerPath({
method: "get",
path: "/api/issues/{id}/external-objects",
+65
View File
@@ -0,0 +1,65 @@
import { and, desc, eq, inArray, lt, lte, or, sql } from "drizzle-orm";
import { linkPreviews, type Db } from "@paperclipai/db";
import type { LinkPreviewResult, LinkPreviewTarget } from "@paperclipai/shared";
import type { LinkPreviewScope } from "./link-previews.js";
export interface StoredLinkPreview { result: LinkPreviewResult; expiresAt: number; checkedAt: number }
export interface LinkPreviewStore {
read(scope: LinkPreviewScope, targets: readonly LinkPreviewTarget[]): Promise<Map<string, StoredLinkPreview>>;
write(scope: LinkPreviewScope, target: LinkPreviewTarget, entry: Omit<StoredLinkPreview, "checkedAt">, checkedAt: number): Promise<void>;
}
const MAX_ROWS_PER_VIEWER = 1_000;
const EXPIRED_RETENTION_MS = 30 * 24 * 60 * 60_000;
export function createLinkPreviewStore(db: Db): LinkPreviewStore {
return {
async read(scope, targets) {
if (!targets.length) return new Map();
// One covering identity-index lookup for the page, never a task join or a provider call.
const rows = await db.select().from(linkPreviews).where(and(
eq(linkPreviews.companyId, scope.companyId), eq(linkPreviews.viewerId, scope.userId ?? ""),
inArray(linkPreviews.canonicalUrl, targets.map(target => target.canonicalUrl)),
));
const byUrl = new Map(targets.map(target => [target.canonicalUrl, target]));
return new Map(rows.flatMap(row => {
const target = byUrl.get(row.canonicalUrl);
if (!target || target.provider !== row.channel || target.kind !== row.resourceKind) return [];
const data = row.metadata;
const preview = row.status !== null && typeof data.title === "string" ? {
target, title: data.title, state: row.status as NonNullable<LinkPreviewResult["preview"]>["state"],
author: typeof data.author === "string" ? data.author : null,
updatedAt: typeof data.updatedAt === "string" ? data.updatedAt : null,
} : null;
return [[row.canonicalUrl, { result: { preview, maxAgeMs: 0 }, expiresAt: row.expiresAt.getTime(), checkedAt: row.checkedAt.getTime() }] as const];
}));
},
async write(scope, target, entry, checkedAt) {
const preview = entry.result.preview;
const values = {
companyId: scope.companyId, viewerId: scope.userId ?? "", channel: target.provider,
resourceKind: target.kind, canonicalUrl: target.canonicalUrl, status: preview?.state ?? null,
metadata: preview ? { title: preview.title, author: preview.author, updatedAt: preview.updatedAt } : {},
checkedAt: new Date(checkedAt), expiresAt: new Date(entry.expiresAt),
};
await db.transaction(async tx => {
// Serialize only this cache scope, including across server replicas.
// The row cap must hold when several distinct URLs are saved together.
const lockKey = JSON.stringify(["link-previews", values.companyId, values.viewerId]);
await tx.execute(sql`SELECT pg_advisory_xact_lock(hashtextextended(${lockKey}, 0))`);
await tx.insert(linkPreviews).values(values).onConflictDoUpdate({
target: [linkPreviews.companyId, linkPreviews.viewerId, linkPreviews.canonicalUrl],
set: values, setWhere: lte(linkPreviews.checkedAt, values.checkedAt),
});
const scopeCondition = and(eq(linkPreviews.companyId, values.companyId), eq(linkPreviews.viewerId, values.viewerId));
// Cleanup runs on writes, without a worker or extra page-read queries.
// The identity index covers this scope, whose retained size is bounded.
await tx.delete(linkPreviews).where(and(scopeCondition, or(
lt(linkPreviews.expiresAt, new Date(checkedAt - EXPIRED_RETENTION_MS)),
inArray(linkPreviews.id, tx.select({ id: linkPreviews.id }).from(linkPreviews).where(scopeCondition)
.orderBy(desc(linkPreviews.checkedAt), desc(linkPreviews.id)).offset(MAX_ROWS_PER_VIEWER)),
)));
});
},
};
}
+96 -14
View File
@@ -1,4 +1,5 @@
import type { LinkPreviewResult, LinkPreviewTarget } from "@paperclipai/shared";
import type { LinkPreviewStore, StoredLinkPreview } from "./link-preview-store.js";
import { recognizeLinkPreview } from "@paperclipai/shared";
export interface LinkPreviewScope { companyId: string; userId: string | null }
@@ -9,22 +10,49 @@ export interface LinkPreviewProvider {
const UNAVAILABLE: LinkPreviewResult = { preview: null, maxAgeMs: 60_000 };
/** Request-driven and process-local. No DB writes, sweep, timers while idle, or cross-viewer reuse. */
/** Request-driven cache: durable across restarts, isolated by viewer, with no idle worker. */
export function createLinkPreviewService(providers: readonly LinkPreviewProvider[], options: {
now?: () => number; timeoutMs?: number; maxEntries?: number; maxConcurrent?: number;
store?: LinkPreviewStore; now?: () => number; timeoutMs?: number; maxEntries?: number; maxConcurrent?: number;
} = {}) {
const now = options.now ?? Date.now;
const cache = new Map<string, { result: LinkPreviewResult; expiresAt: number }>();
const cache = new Map<string, StoredLinkPreview>();
const inFlight = new Map<string, Promise<LinkPreviewResult>>();
let activeOperations = 0;
const byKey = new Map(providers.map(provider => [provider.key, provider]));
function remember(key: string, entry: StoredLinkPreview) {
if ((cache.get(key)?.checkedAt ?? -Infinity) > entry.checkedAt) return;
cache.delete(key);
while (cache.size >= (options.maxEntries ?? 1_000)) cache.delete(cache.keys().next().value!);
cache.set(key, entry);
}
const keyOf = (scope: LinkPreviewScope, url: string) => JSON.stringify([scope.companyId, scope.userId, url]);
const remaining = (entry: StoredLinkPreview) => ({ ...entry.result, maxAgeMs: Math.max(0, entry.expiresAt - now()) });
const newest = (memory: StoredLinkPreview | undefined, stored: StoredLinkPreview | undefined) =>
stored && (!memory || stored.checkedAt >= memory.checkedAt) ? stored : memory;
return {
async cached(scope: LinkPreviewScope, hrefs: readonly string[]) {
const targets = [...new Map(hrefs.map(recognizeLinkPreview).filter(target => target && byKey.has(target.provider))
.map(target => [target!.canonicalUrl, target!])).values()];
const stored = await options.store?.read(scope, targets).catch(() => new Map<string, StoredLinkPreview>());
const results: Record<string, LinkPreviewResult> = {};
for (const target of targets) {
const key = keyOf(scope, target.canonicalUrl);
const memory = cache.get(key);
const entry = newest(memory, stored?.get(target.canonicalUrl));
if (entry) { remember(key, entry); results[target.canonicalUrl] = remaining(entry); }
// A batch miss is not a provider result. A later lookup must still
// check durable storage, which another replica may have populated.
else cache.delete(key);
}
return { results };
},
async get(scope: LinkPreviewScope, href: string): Promise<LinkPreviewResult> {
const target = recognizeLinkPreview(href);
const provider = target && byKey.get(target.provider);
if (!target || !provider) return UNAVAILABLE;
const key = JSON.stringify([scope.companyId, scope.userId, target.canonicalUrl]);
const key = keyOf(scope, target.canonicalUrl);
const cached = cache.get(key);
if (cached && cached.expiresAt > now()) {
return { ...cached.result, maxAgeMs: cached.expiresAt - now() };
@@ -35,20 +63,74 @@ export function createLinkPreviewService(providers: readonly LinkPreviewProvider
const controller = new AbortController();
let timer: ReturnType<typeof setTimeout>;
let previous = cached;
const checkedAt = now();
let providerStarted = false;
let storeWriteStarted = false;
let durableReadSucceeded = !options.store;
let timeoutWrite: Promise<void> | undefined;
const fallback = (): LinkPreviewResult => ({ preview: newest(previous, cache.get(key))?.result.preview ?? null, maxAgeMs: 60_000 });
const deadline = new Promise<LinkPreviewResult>(resolve => {
timer = setTimeout(() => { controller.abort(); resolve(UNAVAILABLE); }, options.timeoutMs ?? 5_000);
timer = setTimeout(() => {
const current = cache.get(key);
if (current && current.checkedAt > checkedAt) {
controller.abort();
resolve(remaining(current));
return;
}
const result = current && current.expiresAt > now() ? remaining(current) : fallback();
const entry = { result, expiresAt: now() + result.maxAgeMs,
checkedAt: durableReadSucceeded || storeWriteStarted ? checkedAt : previous?.checkedAt ?? 0 };
remember(key, entry);
// Publish timeout freshness without delaying the caller. The lookup
// keeps its slot until both the provider and this write settle.
if (options.store && durableReadSucceeded && providerStarted && !storeWriteStarted) {
timeoutWrite = Promise.resolve().then(() => options.store!.write(scope, target, entry, checkedAt)).catch(() => {});
}
controller.abort();
resolve(result);
}, options.timeoutMs ?? 5_000);
});
// Keep an underlying operation in the concurrency budget until it settles,
// even if a credential backend ignores cancellation after our deadline.
// All underlying work, including uncancellable SQL, retains its slot until
// settlement. The caller's deadline does not release the resource budget.
activeOperations++;
const operation = Promise.resolve().then(() => provider.resolve(target, scope, controller.signal))
.catch(() => UNAVAILABLE).finally(() => { activeOperations--; });
const request = Promise.race([operation, deadline]).then(result => {
const operation = Promise.resolve().then(async () => {
if (options.store) {
// Fresh memory returned above. Expired memory is only a fallback:
// another replica may have saved a newer result, even with a shorter TTL.
const stored = await options.store.read(scope, [target]).catch(() => null);
durableReadSucceeded = stored !== null;
previous = newest(newest(previous, cache.get(key)), stored?.get(target.canonicalUrl));
if (!controller.signal.aborted && previous) remember(key, previous);
}
if (controller.signal.aborted) return fallback();
if (previous && previous.expiresAt > now()) { remember(key, previous); return remaining(previous); }
providerStarted = true;
const result = await provider.resolve(target, scope, controller.signal).catch(() => UNAVAILABLE);
if (controller.signal.aborted) return fallback();
// A page batch can hydrate another replica's newer status while the
// provider is pending. That result wins over this older lookup.
previous = newest(previous, cache.get(key));
if (previous && previous.checkedAt > checkedAt) return remaining(previous);
const bounded = { ...result, maxAgeMs: Math.max(1_000, Math.min(result.maxAgeMs, 60 * 60_000)) };
cache.delete(key);
while (cache.size >= (options.maxEntries ?? 1_000)) cache.delete(cache.keys().next().value!);
cache.set(key, { result: bounded, expiresAt: now() + bounded.maxAgeMs });
return bounded;
if (!bounded.preview && previous?.result.preview) bounded.preview = previous.result.preview;
const entry = { result: bounded, expiresAt: now() + bounded.maxAgeMs,
checkedAt: durableReadSucceeded || result.preview ? checkedAt : previous?.checkedAt ?? 0 };
remember(key, entry);
// A failed durable read cannot establish which saved status a provider
// failure would replace. Keep that backoff local until storage recovers.
if (options.store && (durableReadSucceeded || result.preview)) {
storeWriteStarted = true;
await options.store.write(scope, target, entry, checkedAt).catch(() => {});
}
return remaining(entry);
}).catch(fallback).finally(async () => { await timeoutWrite; activeOperations--; });
const request = Promise.race([operation, deadline]).then(result => {
const current = cache.get(key);
// Hydration can also finish while the operation awaits its SQL write.
if (current && current.checkedAt > checkedAt) return remaining(current);
if (!current || current.expiresAt <= now()) remember(key, { result, expiresAt: now() + result.maxAgeMs, checkedAt });
return result;
}).finally(() => {
clearTimeout(timer!);
if (inFlight.get(key) === request) inFlight.delete(key);
@@ -268,7 +268,7 @@
"packages/shared/src/home-paths.ts": "fa4f1c440653b76b3dbb7a51026d557287deb45d",
"packages/shared/src/humanize-connection.test.ts": "ba9e90a44c63452b0bf9f733adb90241921e8eca",
"packages/shared/src/humanize-connection.ts": "fbd414476a0ef7106a7c80d8572208781b091699",
"packages/shared/src/index.ts": "efab7ad18d89a60ddd3f961e35060b68424ef40b",
"packages/shared/src/index.ts": "825867d65bbb0783a2294e1838231e68669a8fc3",
"packages/shared/src/issue-activity-identity.test.ts": "3dc294640f54c2be67943e3e0149cf4eff2c95ca",
"packages/shared/src/issue-activity-identity.ts": "1b0f07ec3fe0506aaaf80a4fa9893f3baf8f0075",
"packages/shared/src/issue-attribution.test.ts": "84563685e7f3c48029243cf1cc24a3a09ee3cde9",
@@ -282,7 +282,7 @@
"packages/shared/src/issue-write-denial.test.ts": "e21b93aa79e6635c019ac338eff954ec618a98ed",
"packages/shared/src/issue-write-denial.ts": "e6b4d72a83f6e4e76e85e34a80f88d1d69b7dfe4",
"packages/shared/src/link-preview.test.ts": "136041c928343ad6272f43701b39cf95be255f3e",
"packages/shared/src/link-preview.ts": "ffa5eb960a7e4bcfa914b4d6029c9fc50f17e027",
"packages/shared/src/link-preview.ts": "b7431d4c30a12067e80e6806728b3ae7c7541f8f",
"packages/shared/src/markdown-work-products.test.ts": "7293d14ad45ec2b11b5deaab4ce070c36f7c9010",
"packages/shared/src/markdown-work-products.ts": "16a88cfb717f325c247a516e78b47d53871b52ed",
"packages/shared/src/mcp-apps.test.ts": "848f3247bdc98af398daa144b0e44991a3f352b3",
@@ -167,3 +167,54 @@ for (const available of [true, false]) {
await expect(links.nth(1)).toHaveAttribute("data-link-state", available ? "open" : "unknown");
});
}
test("page load restores stored status and refreshes stale data on idle without hover", async ({ page }, testInfo) => {
const errors: string[] = [];
page.on("pageerror", error => errors.push(error.message));
await page.addInitScript(() => {
const realFetch = window.fetch.bind(window);
let activeFetch = realFetch;
Object.defineProperty(window, "fetch", {
configurable: true, get: () => activeFetch,
set: (fixtureFetch: typeof fetch) => {
activeFetch = (input, init) => {
const url = input instanceof Request ? input.url : String(input);
return url.includes("/link-preview?") || url.includes("/link-previews/cached") ? realFetch(input, init) : fixtureFetch(input, init);
};
},
});
});
const canonicalUrl = "https://github.com/acme/web/pull/241";
const preview = { target: { provider: "github", kind: "pull_request", canonicalUrl, label: "acme/web#241" },
title: "Status from the database", state: "merged", author: "octocat", updatedAt: null };
let cached = 0, resolved = 0;
let release!: () => void;
const refresh = new Promise<void>(done => { release = done; });
await page.route("**/api/companies/company-storybook/link-previews/cached?*", async route => {
cached++;
expect(new URL(route.request().url()).searchParams.getAll("url")).toEqual([canonicalUrl]);
await route.fulfill({ json: { results: { [canonicalUrl]: { preview, maxAgeMs: 0 } } } });
});
await page.route("**/api/companies/company-storybook/link-preview?*", async route => {
resolved++;
await refresh;
await route.fulfill({ json: { preview: { ...preview, state: "closed" }, maxAgeMs: 60_000 } });
});
await page.goto("/iframe.html?id=tasks-link-previews--page-load&viewMode=story&globals=theme:dark");
await finished(page);
const links = page.locator("a[data-link-state]");
await expect(links).toHaveCount(2);
await expect(links.first()).toHaveAttribute("data-link-state", "merged");
await expect(links.last()).toHaveAttribute("data-link-state", "merged");
await expect(page.getByRole("tooltip")).toHaveCount(0);
await expect.poll(() => resolved).toBe(1);
expect(cached).toBe(1);
release();
await expect(links.first()).toHaveAttribute("data-link-state", "closed");
await expect(links.last()).toHaveAttribute("data-link-state", "closed");
await links.first().focus();
await expect(page.getByRole("tooltip")).toBeVisible();
expect(resolved).toBe(1);
await page.screenshot({ path: testInfo.outputPath("pr-status-from-database-after-idle.png"), fullPage: true });
expect(errors).toEqual([]);
});
+6 -16
View File
@@ -1,12 +1,11 @@
import { useId, useMemo, useRef, useState, type ReactElement } from "react";
import { useEffect, useId, useMemo, useRef, useState, type ReactElement } from "react";
import { useQuery } from "@tanstack/react-query";
import { PreviewCard } from "@base-ui/react/preview-card";
import { GitMerge, GitPullRequest, GitPullRequestClosed, GitPullRequestDraft, TriangleAlert } from "lucide-react";
import type { LinkPreview as Preview, LinkPreviewResult, LinkPreviewState, LinkPreviewTarget } from "@paperclipai/shared";
import { useLinkPreviewContext, type LinkPreviewContextValue } from "@/context/LinkPreviewContext";
import { timeAgo } from "@/lib/timeAgo";
import { retryTransientOnly } from "@/lib/query-client";
import { queryKeys } from "@/lib/queryKeys";
import { linkPreviewQueryOptions } from "@/lib/link-preview-query";
export const linkPreviewStateLabel: Record<LinkPreviewState, string> = {
unknown: "Status unknown", open: "Open", merged: "Merged", conflicts: "Merge conflicts", draft: "Draft", closed: "Closed",
@@ -18,13 +17,13 @@ export const linkPreviewStateIcon = {
function PreviewCardContents({ target, preview, loading }: { target: LinkPreviewTarget; preview?: Preview | null; loading: boolean }) {
const state = preview?.state ?? "unknown";
const Icon = linkPreviewStateIcon[state];
const Icon = linkPreviewStateIcon[state] ?? linkPreviewStateIcon.unknown;
const updated = preview?.updatedAt && Number.isFinite(Date.parse(preview.updatedAt)) ? preview.updatedAt : null;
return <div className="flex min-w-0 flex-col gap-3" data-link-preview-state={state}>
<div className="flex items-center gap-2 text-xs text-muted-foreground">
<Icon aria-hidden="true" className="link-preview-status size-4 shrink-0" data-state={state} />
<span className="min-w-0 break-all font-mono">{target.label}</span>
{preview ? <span className="link-preview-status ml-auto shrink-0" data-state={state}>{linkPreviewStateLabel[state]}</span> : null}
{preview ? <span className="link-preview-status ml-auto shrink-0" data-state={state}>{linkPreviewStateLabel[state] ?? state}</span> : null}
</div>
{preview ? <>
<p className="m-0 text-sm font-medium leading-snug">{preview.title}</p>
@@ -50,17 +49,8 @@ function ConnectedLinkPreview({ context, target, children }: {
getBoundingClientRect: () => triggerRef.current?.getClientRects()[0] ?? new DOMRect(),
get contextElement() { return triggerRef.current ?? undefined; },
}), []);
const query = useQuery({
queryKey: queryKeys.linkPreview(context.companyId, context.userId, target.canonicalUrl),
// Optional enrichment: a failed read is a cached absence, not a global error.
queryFn: async (): Promise<LinkPreviewResult> => {
try { return await context.load(target); } catch { return { preview: null, maxAgeMs: 60_000 }; }
},
enabled: open,
staleTime: query => query.state.data?.maxAgeMs ?? 60_000,
gcTime: 5 * 60_000,
retry: retryTransientOnly(0), refetchOnMount: false, refetchOnWindowFocus: false, refetchOnReconnect: false,
});
useEffect(() => context.register?.(target), [context, target.canonicalUrl]);
const query = useQuery({ ...linkPreviewQueryOptions(context, target), enabled: open });
return <PreviewCard.Root open={open} onOpenChange={setOpen}>
<PreviewCard.Trigger ref={triggerRef} delay={350} closeDelay={150}
onFocus={() => setOpen(true)} onBlur={() => setOpen(false)}
+2 -2
View File
@@ -57,7 +57,7 @@ export function MarkdownUrlLink({ href, children, title, style, knownState, exte
const label = github?.label ? compactUrlLabel(children, href, github.label) : children;
const target = recognizeLinkPreview(href);
const render = (state: LinkPreviewState = knownState ?? "unknown", describedBy?: string, preview?: Preview | null) => {
const StateIcon = target ? linkPreviewStateIcon[state] : Icon;
const StateIcon = target ? linkPreviewStateIcon[state] ?? linkPreviewStateIcon.unknown : Icon;
const leadingIcon = github ? <StateIcon aria-hidden="true" className="mr-1 inline h-3.5 w-3.5 align-(--va-0_125em)" /> : null;
const trailingIcon = external && !github ? <ExternalLink aria-hidden="true" className="ml-1 inline h-3 w-3 align-(--va-0_125em)" /> : null;
return <a
@@ -67,7 +67,7 @@ export function MarkdownUrlLink({ href, children, title, style, knownState, exte
data-external-link={externalStatus ? "resolved" : undefined}
data-external-status={externalStatus?.category}
data-external-liveness={externalStatus?.liveness}
aria-label={preview ? `${target!.label} ${linkPreviewStateLabel[state]}: ${preview.title}` : externalStatus?.ariaLabel}
aria-label={preview ? `${target!.label} ${linkPreviewStateLabel[state] ?? state}: ${preview.title}` : externalStatus?.ariaLabel}
data-link-state={target ? state : undefined}
className={target ? `link-preview-status${externalStatus ? " paperclip-markdown-external-ref" : ""}` : undefined}
data-state={target ? state : undefined}
@@ -0,0 +1,46 @@
// @vitest-environment jsdom
import { act, useEffect } from "react";
import { createRoot, type Root } from "react-dom/client";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { afterEach, expect, it, vi } from "vitest";
import { recognizeLinkPreview } from "@paperclipai/shared";
import { LinkPreviewProvider, useLinkPreviewContext } from "./LinkPreviewContext";
import { queryKeys } from "@/lib/queryKeys";
const target = recognizeLinkPreview("https://github.com/acme/web/pull/42")!;
function MountedLink() {
const context = useLinkPreviewContext();
useEffect(() => context?.register?.(target), [context]);
return <span>Task content is ready</span>;
}
let root: Root | null = null;
afterEach(() => { if (root) act(() => root!.unmount()); root = null; document.body.replaceChildren(); vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllGlobals(); });
it("aborts a stalled batch GET at its deadline and continues idle enrichment", async () => {
vi.useFakeTimers();
vi.stubGlobal("IS_REACT_ACT_ENVIRONMENT", true);
vi.spyOn(document, "readyState", "get").mockReturnValue("complete");
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
let aborted = false;
const fetch = vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => {
if (String(input).includes("/link-previews/cached")) return new Promise<Response>((_resolve, reject) => {
init?.signal?.addEventListener("abort", () => { aborted = true; reject(new DOMException("Aborted", "AbortError")); });
});
return Response.json({ preview: { target, title: "Loaded on idle", state: "merged", author: null, updatedAt: null }, maxAgeMs: 60_000 });
});
vi.stubGlobal("fetch", fetch);
vi.stubGlobal("requestIdleCallback", (callback: () => void) => setTimeout(callback, 0));
vi.stubGlobal("cancelIdleCallback", (id: number) => clearTimeout(id));
const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
const host = document.createElement("div"); document.body.append(host);
root = createRoot(host);
await act(async () => { root!.render(<QueryClientProvider client={client}><LinkPreviewProvider companyId="company-a" userId="alice"><MountedLink /></LinkPreviewProvider></QueryClientProvider>); });
expect(host.textContent).toContain("Task content is ready");
await act(async () => { await vi.advanceTimersByTimeAsync(1); });
expect(fetch).toHaveBeenCalledTimes(1);
await act(async () => { await vi.advanceTimersByTimeAsync(6_600); });
expect(aborted).toBe(true);
expect(fetch).toHaveBeenCalledTimes(2);
expect(client.getQueryData<{ preview: { state: string } }>(queryKeys.linkPreview("company-a", "alice", target.canonicalUrl))?.preview.state).toBe("merged");
act(() => root!.unmount()); root = null; client.clear();
});
+23 -5
View File
@@ -1,25 +1,43 @@
import { createContext, useContext, useMemo, type ReactNode } from "react";
import type { LinkPreviewResult, LinkPreviewTarget } from "@paperclipai/shared";
import { createContext, useContext, useMemo, useLayoutEffect, useRef, type ReactNode } from "react";
import { useQueryClient } from "@tanstack/react-query";
import { createLinkPreviewPageLoader } from "@/lib/link-preview-page-loader";
import type { LinkPreviewBatchResult, LinkPreviewResult, LinkPreviewTarget } from "@paperclipai/shared";
import { api } from "@/api/client";
export interface LinkPreviewContextValue {
companyId: string;
userId: string | null;
register?: (target: LinkPreviewTarget) => () => void;
load: (target: LinkPreviewTarget) => Promise<LinkPreviewResult>;
}
const LinkPreviewContext = createContext<LinkPreviewContextValue | null>(null);
export function LinkPreviewProvider({ companyId, userId, children }: {
companyId: string | null; userId: string | null; children: ReactNode;
export function LinkPreviewProvider({ companyId, userId, children, autoLoad = true }: {
companyId: string | null; userId: string | null; children: ReactNode; autoLoad?: boolean;
}) {
const client = useQueryClient();
const pageLoader = useRef<ReturnType<typeof createLinkPreviewPageLoader> | null>(null);
const value = useMemo<LinkPreviewContextValue | null>(() => companyId ? ({
companyId, userId,
register: target => pageLoader.current?.register(target) ?? (() => {}),
load: target => api.get<LinkPreviewResult>(
`/companies/${companyId}/link-preview?${new URLSearchParams({ url: target.canonicalUrl })}`,
{ timeoutMs: 6_000, cache: "no-store" },
),
}) : null, [companyId, userId]);
}) : null, [companyId, userId, autoLoad]);
// Link registration uses passive effects, after this layout effect initializes the registry.
useLayoutEffect(() => {
if (!value || !autoLoad) return;
const loader = createLinkPreviewPageLoader(client, {
...value,
readCached: targets => api.get<LinkPreviewBatchResult>(`/companies/${companyId}/link-previews/cached?${
new URLSearchParams(targets.map(target => ["url", target.canonicalUrl]))
}`, { timeoutMs: 6_000, cache: "no-store" }),
});
pageLoader.current = loader;
return () => { loader.dispose(); if (pageLoader.current === loader) pageLoader.current = null; };
}, [value, autoLoad, client, companyId]);
return <LinkPreviewContext.Provider value={value}>{children}</LinkPreviewContext.Provider>;
}
+130
View File
@@ -0,0 +1,130 @@
import { QueryClient } from "@tanstack/react-query";
import { recognizeLinkPreview, type LinkPreviewResult, type LinkPreviewTarget } from "@paperclipai/shared";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { createLinkPreviewPageLoader, MAX_PAGE_LINK_PREVIEWS, type LinkPreviewPageEnvironment } from "./link-preview-page-loader";
import { linkPreviewQueryOptions } from "./link-preview-query";
const target = recognizeLinkPreview("https://github.com/acme/web/pull/42")!;
const result: LinkPreviewResult = { preview: { target, title: "Saved status", state: "merged", author: null, updatedAt: null }, maxAgeMs: 60_000 };
const disposers: Array<() => void> = [];
function fixture() {
const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
let loaded = false;
let onLoad = () => {};
const idle = new Set<() => void>();
const environment: LinkPreviewPageEnvironment = {
loaded: () => loaded, onLoad: cb => { onLoad = cb; return () => { onLoad = () => {}; }; },
idle: cb => { idle.add(cb); return () => { idle.delete(cb); }; },
};
const context = { companyId: "company-a", userId: "alice", load: vi.fn(async () => result), readCached: vi.fn(async (_targets: LinkPreviewTarget[]) => ({ results: {} as Record<string, LinkPreviewResult> })) };
const loader = createLinkPreviewPageLoader(client, context, environment);
disposers.push(() => { loader.dispose(); client.clear(); });
return { client, context, loader, idle,
loaded() { loaded = true; onLoad(); },
runIdle() { const callbacks = [...idle]; idle.clear(); callbacks.forEach(cb => cb()); },
};
}
async function settle() { await vi.advanceTimersByTimeAsync(1); }
beforeEach(() => vi.useFakeTimers());
afterEach(() => { disposers.splice(0).forEach(dispose => dispose()); vi.useRealTimers(); });
describe("page link enrichment", () => {
it("hydrates duplicate links in one batch before load, then reuses fresh status without a provider request", async () => {
const f = fixture();
f.context.readCached.mockResolvedValue({ results: { [target.canonicalUrl]: result } });
for (let i = 0; i < 500; i++) f.loader.register(target);
expect(f.context.readCached).not.toHaveBeenCalled();
await settle();
expect(f.context.readCached).toHaveBeenCalledExactlyOnceWith([target]);
expect(f.client.getQueryData(linkPreviewQueryOptions(f.context, target).queryKey)).toEqual(result);
expect(f.idle.size).toBe(0);
f.loaded(); f.runIdle(); await settle();
expect(f.context.load).not.toHaveBeenCalled();
});
it("waits for core page queries and load, then resolves without hover and stays quiet", async () => {
const f = fixture();
let finish!: (value: number) => void;
const core = f.client.fetchQuery({ queryKey: ["issues", "detail", "42"], queryFn: () => new Promise<number>(done => { finish = done; }) });
f.loader.register(target); f.loaded(); await settle();
expect(f.context.load).not.toHaveBeenCalled(); expect(f.idle.size).toBe(0);
finish(1); await core; expect(f.idle.size).toBe(1);
f.runIdle(); await settle(); expect(f.context.load).toHaveBeenCalledTimes(1);
await vi.advanceTimersByTimeAsync(120_000); f.loaded(); f.runIdle(); await settle();
expect(f.context.load).toHaveBeenCalledTimes(1);
});
it("shows stale saved status and coalesces an overlapping hover/idle lookup", async () => {
const f = fixture();
f.context.readCached.mockResolvedValue({ results: { [target.canonicalUrl]: { ...result, maxAgeMs: 0 } } });
let finish!: (value: LinkPreviewResult) => void;
f.context.load.mockImplementation(() => new Promise(done => { finish = done; }));
f.loader.register(target); f.loaded(); await settle();
expect(f.client.getQueryData<LinkPreviewResult>(linkPreviewQueryOptions(f.context, target).queryKey)?.preview?.state).toBe("merged");
const hover = f.client.fetchQuery(linkPreviewQueryOptions(f.context, target));
f.runIdle(); await settle(); expect(f.context.load).toHaveBeenCalledTimes(1);
finish(result); await hover; await settle(); expect(f.idle.size).toBe(0);
});
it("does not replace newer hover status with a slower cache read", async () => {
const f = fixture();
let finish!: (value: { results: Record<string, LinkPreviewResult> }) => void;
f.context.readCached.mockImplementation(() => new Promise(done => { finish = done; }));
f.loader.register(target); await settle();
await f.client.fetchQuery(linkPreviewQueryOptions(f.context, target));
finish({ results: { [target.canonicalUrl]: { ...result, preview: { ...result.preview!, state: "open" }, maxAgeMs: 0 } } });
await settle();
expect(f.client.getQueryData<LinkPreviewResult>(linkPreviewQueryOptions(f.context, target).queryKey)?.preview?.state).toBe("merged");
});
it("bounds a large page, batches SQL cache reads, and limits provider concurrency", async () => {
const f = fixture();
f.context.load.mockImplementation(() => new Promise(() => {}));
for (let i = 0; i < 1000; i++) f.loader.register(recognizeLinkPreview(target.canonicalUrl + i)!);
await vi.advanceTimersByTimeAsync(10);
expect(f.context.readCached.mock.calls.flatMap(([targets]) => targets)).toHaveLength(MAX_PAGE_LINK_PREVIEWS);
expect(f.context.readCached.mock.calls.every(([targets]) => targets.length <= 50)).toBe(true);
f.loaded(); f.runIdle(); await settle();
expect(f.context.load).toHaveBeenCalledTimes(3);
expect(f.idle.size).toBe(0);
});
it("removes departed links and ignores late hydration after navigation or account change", async () => {
const f = fixture();
const unregister = f.loader.register(target);
unregister(); f.loaded(); await settle(); f.runIdle();
expect(f.context.readCached).not.toHaveBeenCalled(); expect(f.context.load).not.toHaveBeenCalled();
let finish!: (value: { results: Record<string, LinkPreviewResult> }) => void;
f.context.readCached.mockImplementation(() => new Promise(done => { finish = done; }));
f.loader.register(target); await settle(); f.loader.dispose();
finish({ results: { [target.canonicalUrl]: result } }); await settle();
expect(f.client.getQueryData(linkPreviewQueryOptions(f.context, target).queryKey)).toBeUndefined();
});
it("shares a pending database read when the same URL remounts", async () => {
const f = fixture();
let finish!: (value: { results: Record<string, LinkPreviewResult> }) => void;
f.context.readCached.mockImplementation(() => new Promise(done => { finish = done; }));
const unregister = f.loader.register(target); await settle();
unregister(); f.loader.register(target); await settle();
finish({ results: { [target.canonicalUrl]: result } }); await settle();
expect(f.context.readCached).toHaveBeenCalledTimes(1);
expect(f.client.getQueryData(linkPreviewQueryOptions(f.context, target).queryKey)).toEqual(result);
});
it("retains saved status when automatic refresh has a network error", async () => {
const f = fixture();
f.client.setQueryData(linkPreviewQueryOptions(f.context, target).queryKey, { ...result, maxAgeMs: 0 });
f.context.load.mockRejectedValue(new Error("network unavailable"));
f.loader.register(target); f.loaded(); await settle(); f.runIdle(); await settle();
expect(f.client.getQueryData<LinkPreviewResult>(linkPreviewQueryOptions(f.context, target).queryKey)?.preview).toEqual(result.preview);
expect(f.context.readCached).not.toHaveBeenCalled();
});
it("bounds encoded batch URLs for long resource names", async () => {
const f = fixture();
for (let i = 0; i < 100; i++) f.loader.register(recognizeLinkPreview(`https://github.com/acme/${"a".repeat(1900)}/pull/${i + 1}`)!);
await vi.advanceTimersByTimeAsync(100);
expect(f.context.readCached.mock.calls.flatMap(([targets]) => targets)).toHaveLength(100);
expect(f.context.readCached.mock.calls.every(([targets]) => targets.reduce((n, target) => n + encodeURIComponent(target.canonicalUrl).length + 5, 0) <= 6_000)).toBe(true);
});
it("keeps failures quiet and lets hover work when cached hydration fails", async () => {
const f = fixture();
f.context.readCached.mockRejectedValue(new Error("cache unavailable"));
f.loader.register(target); f.loaded(); await settle(); f.runIdle(); await settle();
expect(f.context.load).toHaveBeenCalledTimes(1);
expect(f.client.getQueryData(linkPreviewQueryOptions(f.context, target).queryKey)).toEqual(result);
});
});
+139
View File
@@ -0,0 +1,139 @@
import type { QueryClient } from "@tanstack/react-query";
import { LINK_PREVIEW_BATCH_LIMIT, type LinkPreviewBatchResult, type LinkPreviewResult, type LinkPreviewTarget } from "@paperclipai/shared";
import { linkPreviewQueryOptions, type LinkPreviewLoader } from "./link-preview-query";
export const MAX_PAGE_LINK_PREVIEWS = 200;
const PAGE_CONCURRENCY = 3;
export interface LinkPreviewPageEnvironment {
loaded(): boolean;
onLoad(callback: () => void): () => void;
idle(callback: () => void): () => void;
}
export const browserLinkPreviewEnvironment: LinkPreviewPageEnvironment = {
loaded: () => document.readyState === "complete" && document.visibilityState !== "hidden",
onLoad: callback => {
window.addEventListener("load", callback);
document.addEventListener("visibilitychange", callback);
return () => { window.removeEventListener("load", callback); document.removeEventListener("visibilitychange", callback); };
},
idle: callback => {
// Let input, paint and the initial task requests finish. This is a one-shot
// page event, not a recurring refresh loop.
let cancel = () => {};
const timer = setTimeout(() => {
if (window.requestIdleCallback) {
const id = window.requestIdleCallback(callback, { timeout: 2_000 });
cancel = () => window.cancelIdleCallback(id);
} else {
const id = setTimeout(callback, 0);
cancel = () => clearTimeout(id);
}
}, 500);
return () => { clearTimeout(timer); cancel(); };
},
};
/** One registry per authenticated company. Work is proportional to mounted unique links. */
export function createLinkPreviewPageLoader(client: QueryClient, context: LinkPreviewLoader & {
readCached(targets: LinkPreviewTarget[]): Promise<LinkPreviewBatchResult>;
}, environment: LinkPreviewPageEnvironment = browserLinkPreviewEnvironment) {
const links = new Map<string, { target: LinkPreviewTarget; count: number; hydrated: boolean; attempted: boolean }>();
let disposed = false;
let hydrationTimer: ReturnType<typeof setTimeout> | undefined;
let hydrating = false;
let active = 0;
let cancelIdle: (() => void) | undefined;
const busy = () => client.isFetching({ predicate: query => query.queryKey[0] !== "link-preview" }) > 0;
function scheduleIdle() {
if (disposed || cancelIdle || hydrating || active >= PAGE_CONCURRENCY || !environment.loaded() || busy()
|| ![...links.values()].some(link => link.hydrated && !link.attempted)) return;
cancelIdle = environment.idle(() => {
cancelIdle = undefined;
if (disposed || !environment.loaded() || busy()) return;
for (const link of links.values()) {
if (!link.hydrated || link.attempted) continue;
if (active >= PAGE_CONCURRENCY) break;
link.attempted = true;
active++;
void client.fetchQuery(linkPreviewQueryOptions(context, link.target)).catch(() => {}).finally(() => { active--; scheduleIdle(); });
}
});
}
function scheduleHydration() {
if (disposed || hydrating || hydrationTimer !== undefined) return;
hydrationTimer = setTimeout(() => {
hydrationTimer = undefined;
for (const link of links.values()) {
if (!link.hydrated && client.getQueryData(linkPreviewQueryOptions(context, link.target).queryKey) !== undefined) link.hydrated = true;
}
const batch: Array<{ target: LinkPreviewTarget; count: number; hydrated: boolean; attempted: boolean }> = [];
let bytes = 0;
for (const link of links.values()) {
if (link.hydrated) continue;
const size = encodeURIComponent(link.target.canonicalUrl).length + 5;
if (batch.length >= LINK_PREVIEW_BATCH_LIMIT || bytes + size > 6_000) break;
batch.push(link); bytes += size;
}
if (!batch.length) { scheduleIdle(); return; }
hydrating = true;
const startedAt = Date.now();
for (const link of batch) link.hydrated = true;
const targets = batch.map(link => link.target);
// A renderer can remount while a batch is pending. Keep the read on the
// shared client so both lifetimes reuse the same HTTP request.
void client.fetchQuery({
queryKey: ["link-preview", context.companyId, context.userId, "cached-batch", targets.map(target => target.canonicalUrl).sort()],
queryFn: () => context.readCached(targets), staleTime: 0, gcTime: 0,
retry: false, // query-error-ok: Optional hydration must settle within one deadline so idle enrichment can continue when the cache is unavailable.
}).then(({ results }) => {
if (disposed) return;
for (const link of batch) {
const mounted = links.get(link.target.canonicalUrl);
if (!mounted) continue;
mounted.hydrated = true;
const options = linkPreviewQueryOptions(context, link.target);
const state = client.getQueryState<LinkPreviewResult>(options.queryKey);
const saved = results[link.target.canonicalUrl];
// Never replace a hover result that arrived during the database read.
if (saved && state?.fetchStatus !== "fetching" && (!state || state.dataUpdatedAt <= startedAt)
&& (!state?.data || state.dataUpdatedAt + state.data.maxAgeMs <= Date.now() + saved.maxAgeMs)) {
client.setQueryData(options.queryKey, saved);
}
}
}).catch(() => {}).finally(() => {
hydrating = false;
scheduleHydration();
scheduleIdle();
});
}, 0);
}
const unsubscribe = client.getQueryCache().subscribe(event => {
if (event.query.queryKey[0] !== "link-preview") scheduleIdle();
});
const offLoad = environment.onLoad(scheduleIdle);
return {
register(target: LinkPreviewTarget) {
let link = links.get(target.canonicalUrl);
if (!link && links.size < MAX_PAGE_LINK_PREVIEWS) {
link = { target, count: 0, hydrated: false, attempted: false };
links.set(target.canonicalUrl, link);
}
if (!link) return () => {};
link.count++;
scheduleHydration();
return () => {
if (--link.count === 0) links.delete(target.canonicalUrl);
if (!links.size) { cancelIdle?.(); cancelIdle = undefined; }
};
},
dispose() {
disposed = true;
clearTimeout(hydrationTimer);
cancelIdle?.();
unsubscribe(); offLoad(); links.clear();
},
};
}
+24
View File
@@ -0,0 +1,24 @@
import type { QueryFunctionContext } from "@tanstack/react-query";
import type { LinkPreviewResult, LinkPreviewTarget } from "@paperclipai/shared";
import { queryKeys } from "./queryKeys";
export interface LinkPreviewLoader {
companyId: string;
userId: string | null;
load: (target: LinkPreviewTarget) => Promise<LinkPreviewResult>;
}
/** Hover and deferred page work must share both the key and the query function. */
export function linkPreviewQueryOptions(context: LinkPreviewLoader, target: LinkPreviewTarget) {
return {
queryKey: queryKeys.linkPreview(context.companyId, context.userId, target.canonicalUrl),
queryFn: async ({ client, queryKey }: QueryFunctionContext): Promise<LinkPreviewResult> => {
try { return await context.load(target); } catch { return { preview: client.getQueryData<LinkPreviewResult>(queryKey)?.preview ?? null, maxAgeMs: 60_000 }; }
},
staleTime: (query: { state: { data?: LinkPreviewResult } }) => query.state.data?.maxAgeMs ?? 60_000,
gcTime: 5 * 60_000,
retry: false as const, // query-error-ok: Optional previews preserve saved status and cache failure backoff; automatic retries would exceed the lookup budget.
refetchOnMount: false as const,
refetchOnWindowFocus: false as const, refetchOnReconnect: false as const,
};
}
@@ -8,13 +8,13 @@ import { startPersistedQueryCache } from "@/lib/persisted-query-cache";
import { queryKeys } from "@/lib/queryKeys";
import { prepareFixtureApi } from "../../support/capability-gaps/api";
interface PreviewArgs { state: LinkPreviewState; mode: "ready" | "loading" | "unavailable" | "failure"; longTitle: boolean }
interface PreviewArgs { state: LinkPreviewState; mode: "ready" | "loading" | "unavailable" | "failure"; longTitle: boolean; autoLoad?: boolean }
function PreviewExample({ client: restoredClient }: PreviewArgs & { client?: QueryClient }) {
function PreviewExample({ client: restoredClient, autoLoad = false }: PreviewArgs & { client?: QueryClient }) {
const [localClient] = useState(() => new QueryClient());
const client = restoredClient ?? localClient;
return <QueryClientProvider client={client}>
<LinkPreviewProvider companyId="company-storybook" userId="user-board">
<LinkPreviewProvider companyId="company-storybook" userId="user-board" autoLoad={autoLoad}>
<main className="mx-auto flex max-w-3xl flex-col gap-8 p-6">
<header className="flex flex-col gap-2">
<h1 className="text-xl font-medium">Pull request links</h1>
@@ -62,6 +62,9 @@ const meta = {
},
maxAgeMs: 60_000,
}),
}, {
path: "/api/companies/company-storybook/link-previews/cached", method: "GET",
read: () => ({ results: {} }),
}]);
return fixture.cleanup;
},
@@ -80,6 +83,10 @@ export const ReloadCache: Story = {
render: args => <PersistedPreviewExample {...args} />,
parameters: { paperclip: { notes: "Production browser read-cache persistence with fixture account/company identity. Load a PR, then reload this iframe to retain the learned status. Fresh previews reuse their original read time; stale previews refresh only on hover or focus. No live provider authorization is claimed." } },
};
export const PageLoad: Story = {
args: { autoLoad: true },
parameters: { paperclip: { notes: "Production page registry reads a batch of saved status, then resolves missing previews after document load and core queries settle. Duplicate mounted links share one query. Cache/provider responses are fixtures; database behavior has separate integration tests." } },
};
export const Open: Story = {};
export const Merged: Story = { args: { state: "merged" } };
export const Conflicts: Story = { args: { state: "conflicts" } };