Files
PaperClipAI/ui/src/lib/issue-output.ts
T
DottaandPaperclip 1ccae464c5 feat(ui): add task artifact media gallery and full-row links (#13825)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks collect the files and work products that agents create.
> - The Artifacts tab shows these outputs in rows, which makes videos
hard to compare.
> - Small text links also make artifact rows harder to open.
> - This pull request adds image and video tiles with previews and makes
the full artifact row clickable.
> - Users can compare outputs and open the existing media viewer with
one click.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The task Artifacts tab and media previews in task chat.

**Current behavior**

Video outputs appear as file rows or icons. Users must click a small
link to open a work product. A task with eight video outputs gives
little visual context.

**Proposed behavior**

Show images and videos in a responsive gallery. Show a paused video
frame as the thumbnail. Keep documents, links, and other files in rows
whose entire area opens the item.

**Reason and benefit**

Users can compare generated media without opening each item. Larger
click targets also make the sidebar easier to use.

**Breaking changes**

None. This uses the existing artifact URLs, media viewer, run grouping,
and attachment filters. No API or database changes.

Related work: #11226 added the task sidebar output surface, and #7361
added rich attachment previews. #3524 concerns a separate
reviewed-assets panel. This PR improves the existing task artifact
components. The duplicate search found no active PR for this change.
This is polish for the shipped Artifacts & Work Products roadmap item.

## What Changed

- Add a shared media tile for work products and agent attachments.
- Reuse video and image previews in task artifacts and chat. Seek up to
one second into videos and reset preview state when the source changes.
- Use the existing task gallery for playback and downloads. Preserve
grouping and attachment deduplication.
- Extend native links and buttons across work-product rows, including
keyboard focus indicators.
- Add eight offline Storybook examples for video outputs, mixed media,
clickable rows, narrow and wide panels, missing previews, empty state,
and light mode.
- Register the component in the design guide and document its use.

## Verification

- 108 focused component tests pass, including thumbnail seeking, source
changes, gallery activation, and attachment deduplication.
- `pnpm build`, `pnpm -r typecheck`, Storybook build, token gates, and
`git diff --check` pass locally.
- Reviewed the production components in the embedded browser. Checked
all eight video thumbnails, mixed media, narrow layout, light mode,
blank-area row clicks, keyboard gallery activation for generic-MIME
images, and playback from chat video thumbnails.
- Storybook: open **Tasks / Artifact Gallery** and select **Eight Video
Outputs**, **Mixed Media And Files**, or **Whole Row Clickable**. The
small local clips are synthetic fixtures.
- All build, typecheck, unit, runner, and end-to-end CI jobs pass for
`b8ace289b5e07df5b9f2c319b159f3923ce427b4`. Greptile gives 5/5 with both
review findings resolved. All 54 PR checks pass, including the external
security scan. The full test suite passed in CI. The duplicate serial
local test run was stopped after CI finished; the 108 focused tests,
full build, and recursive typecheck passed locally.

## Risks

- Video thumbnails require the browser to load metadata and a frame. A
slow server or unsupported codec can leave the fallback visible; opening
and downloading still use the existing viewer.
- Full-row click targets change pointer interaction with work-product
cards. Native link and button semantics remain in place.

## Model Used

OpenAI GPT-6 in Codex, with code execution and browser tools. The exact
deployment ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-22 15:03:26 -05:00

327 lines
10 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import {
attachmentArtifactWorkProductMetadataSchema,
type AttachmentArtifactWorkProductMetadata,
type IssueWorkProduct,
} from "@paperclipai/shared";
/**
* Helpers + selectors for the issue Output surface (PAP-10162 Phase 3).
*
* The Output surface promotes attachment-backed artifact work products to a
* first-class slot on the issue page so cloud users can watch / download files
* an agent produced without digging through comments or the host filesystem.
*/
export type OutputFileTone = "video" | "pdf" | "zip" | "image" | "bin";
export interface OutputFileGlyph {
/** Short (≤4 char) label for the file-type tile, e.g. "MP4". */
label: string;
tone: OutputFileTone;
}
/**
* Format a byte count for display.
*
* Examples: `0 B`, `512 B`, `412 KB`, `18.4 MB`, `1.2 GB`. One decimal place is
* used from KB upward, with a trailing `.0` trimmed so round values stay clean.
*/
export function formatBytes(bytes: number): string {
if (!Number.isFinite(bytes) || bytes < 0) return "0 B";
if (bytes < 1024) return `${Math.round(bytes)} B`;
const units = ["KB", "MB", "GB", "TB"];
let value = bytes / 1024;
let unitIndex = 0;
while (value >= 1024 && unitIndex < units.length - 1) {
value /= 1024;
unitIndex += 1;
}
const rounded = value.toFixed(1);
const trimmed = rounded.endsWith(".0") ? rounded.slice(0, -2) : rounded;
return `${trimmed} ${units[unitIndex]}`;
}
/**
* Format a duration in seconds as `m:ss` (under an hour) or `h:mm:ss`.
* Examples: `0:58`, `1:42:09`.
*/
export function formatDuration(seconds: number): string {
if (!Number.isFinite(seconds) || seconds < 0) return "0:00";
const total = Math.floor(seconds);
const hrs = Math.floor(total / 3600);
const mins = Math.floor((total % 3600) / 60);
const secs = total % 60;
const pad = (n: number) => String(n).padStart(2, "0");
if (hrs > 0) {
return `${hrs}:${pad(mins)}:${pad(secs)}`;
}
return `${mins}:${pad(secs)}`;
}
const GENERIC_BINARY_CONTENT_TYPES = new Set([
"application/octet-stream",
"binary/octet-stream",
"application/x-binary",
]);
const VIDEO_FILENAME_EXTENSIONS = [
".mp4",
".m4v",
".webm",
".mov",
".qt",
".quicktime",
];
const IMAGE_FILENAME_EXTENSIONS = [".png", ".jpg", ".jpeg", ".gif", ".webp", ".avif", ".bmp", ".svg", ".ico"];
const BINARY_OUTPUT_APPLICATION_TYPES = new Set([
"application/wasm",
]);
const ZIP_CONTENT_TYPES = new Set([
"application/zip",
"application/x-zip",
"application/x-zip-compressed",
]);
const MARKDOWN_CONTENT_TYPES = new Set([
"text/markdown",
"text/x-markdown",
"application/markdown",
"application/x-markdown",
]);
const DOCUMENT_LIKE_APPLICATION_TYPES = new Set([
"application/csv",
"application/ecmascript",
"application/graphql",
"application/html",
"application/javascript",
"application/json",
"application/ld+json",
"application/sql",
"application/toml",
"application/x-httpd-php",
"application/x-javascript",
"application/x-python",
"application/x-sh",
"application/x-yaml",
"application/xhtml+xml",
"application/xml",
"application/yaml",
]);
const DOCUMENT_LIKE_FILENAME_EXTENSIONS = [
".csv",
".css",
".graphql",
".htm",
".html",
".js",
".json",
".jsx",
".log",
".markdown",
".md",
".mdx",
".php",
".py",
".sh",
".sql",
".toml",
".ts",
".tsx",
".txt",
".xml",
".yaml",
".yml",
];
export function normalizeOutputContentType(contentType: string | null | undefined): string {
return (contentType ?? "").toLowerCase().split(";")[0]?.trim() ?? "";
}
function hasDocumentLikeFilename(originalFilename: string | null | undefined): boolean {
const filename = (originalFilename ?? "").trim().toLowerCase();
if (!filename) return false;
return DOCUMENT_LIKE_FILENAME_EXTENSIONS.some((extension) => filename.endsWith(extension));
}
function isZipContentType(contentType: string): boolean {
return ZIP_CONTENT_TYPES.has(contentType) || contentType.endsWith("+zip");
}
function isDocumentLikeOutputContentType(contentType: string): boolean {
if (!contentType) return false;
if (contentType.startsWith("text/")) return true;
if (MARKDOWN_CONTENT_TYPES.has(contentType)) return true;
if (DOCUMENT_LIKE_APPLICATION_TYPES.has(contentType)) return true;
if (contentType.startsWith("application/") && (contentType.endsWith("+json") || contentType.endsWith("+xml"))) return true;
return false;
}
export function isOutputEligibleContentType(
contentType: string | null | undefined,
originalFilename?: string | null | undefined,
): boolean {
const type = normalizeOutputContentType(contentType);
if (!type) return false;
if (isDocumentLikeOutputContentType(type)) return false;
if (GENERIC_BINARY_CONTENT_TYPES.has(type) && hasDocumentLikeFilename(originalFilename)) return false;
return (
type.startsWith("video/") ||
type.startsWith("image/") ||
type === "application/pdf" ||
BINARY_OUTPUT_APPLICATION_TYPES.has(type) ||
isZipContentType(type) ||
GENERIC_BINARY_CONTENT_TYPES.has(type)
);
}
/**
* Map a MIME type to a short label + tone for the 32×32 file-type tile.
*/
export function getOutputFileGlyph(contentType: string | null | undefined): OutputFileGlyph {
const type = normalizeOutputContentType(contentType);
if (type.startsWith("video/")) {
const subtype = type.slice("video/".length);
if (subtype === "quicktime") return { label: "MOV", tone: "video" };
return { label: (subtype || "vid").toUpperCase().slice(0, 4), tone: "video" };
}
if (type === "application/pdf") return { label: "PDF", tone: "pdf" };
if (isZipContentType(type)) {
return { label: "ZIP", tone: "zip" };
}
if (type.startsWith("image/")) return { label: "IMG", tone: "image" };
if (MARKDOWN_CONTENT_TYPES.has(type)) return { label: "MD", tone: "bin" };
if (type === "text/plain") return { label: "TXT", tone: "bin" };
if (type === "text/csv" || type === "application/csv") return { label: "CSV", tone: "bin" };
if (type === "text/html" || type === "application/html" || type === "application/xhtml+xml") {
return { label: "HTML", tone: "bin" };
}
if (type === "application/json" || type.endsWith("+json")) return { label: "JSON", tone: "bin" };
if (type === "application/xml" || type === "text/xml" || type.endsWith("+xml")) {
return { label: "XML", tone: "bin" };
}
if (type === "application/wasm") return { label: "WASM", tone: "bin" };
return { label: "BIN", tone: "bin" };
}
export function isVideoContentType(contentType: string | null | undefined): boolean {
return normalizeOutputContentType(contentType).startsWith("video/");
}
export function isVideoLikeOutput(
contentType: string | null | undefined,
originalFilename?: string | null | undefined,
): boolean {
const type = normalizeOutputContentType(contentType);
if (type.startsWith("video/")) return true;
if (!GENERIC_BINARY_CONTENT_TYPES.has(type)) return false;
const filename = (originalFilename ?? "").trim().toLowerCase();
return VIDEO_FILENAME_EXTENSIONS.some((extension) => filename.endsWith(extension));
}
export function isImageContentType(contentType: string | null | undefined): boolean {
return normalizeOutputContentType(contentType).startsWith("image/");
}
/** Recognize image uploads with missing or generic MIME types by filename. */
export function isImageLikeOutput(
contentType: string | null | undefined,
originalFilename?: string | null,
): boolean {
const type = normalizeOutputContentType(contentType);
if (isImageContentType(type)) return true;
if (type && !GENERIC_BINARY_CONTENT_TYPES.has(type)) return false;
const filename = (originalFilename ?? "").trim().toLowerCase();
return IMAGE_FILENAME_EXTENSIONS.some((extension) => filename.endsWith(extension));
}
/**
* A single rendered output. `metadata` is null when the work product's stored
* metadata fails validation — the row is still surfaced (degraded) so we never
* silently drop an artifact the agent reported producing.
*/
export interface IssueOutputItem {
id: string;
title: string;
status: string;
isPrimary: boolean;
createdAt: string | Date;
metadata: AttachmentArtifactWorkProductMetadata | null;
/** True when stored metadata could not be parsed into a usable artifact. */
degraded: boolean;
workProduct: IssueWorkProduct;
}
export interface IssueOutputs {
items: IssueOutputItem[];
primary: IssueOutputItem | null;
rest: IssueOutputItem[];
count: number;
}
function toTime(value: string | Date): number {
const t = new Date(value).getTime();
return Number.isNaN(t) ? 0 : t;
}
/**
* Parse attachment-backed artifact work products into renderable outputs.
*
* - Only `type: "artifact"` work products are considered outputs.
* - Metadata is validated against the shared contract; invalid metadata yields
* a degraded item rather than an exception.
* - Ordering: the explicit primary (or the most-recent artifact when none is
* marked primary) comes first, then remaining artifacts by most-recent.
*/
export function getIssueOutputs(workProducts: IssueWorkProduct[] | null | undefined): IssueOutputs {
const artifacts = (workProducts ?? []).filter((wp) => wp.type === "artifact" && wp.provider === "paperclip");
const items: IssueOutputItem[] = artifacts.flatMap((wp) => {
const parsed = attachmentArtifactWorkProductMetadataSchema.safeParse(wp.metadata);
if (parsed.success && !isOutputEligibleContentType(parsed.data.contentType, parsed.data.originalFilename)) {
return [];
}
return [{
id: wp.id,
title: wp.title,
status: typeof wp.status === "string" ? wp.status : "active",
isPrimary: Boolean(wp.isPrimary),
createdAt: wp.createdAt,
metadata: parsed.success ? parsed.data : null,
degraded: !parsed.success,
workProduct: wp,
}];
});
items.sort((a, b) => {
if (a.isPrimary !== b.isPrimary) return a.isPrimary ? -1 : 1;
return toTime(b.createdAt) - toTime(a.createdAt);
});
return {
items,
primary: items[0] ?? null,
rest: items.slice(1),
count: items.length,
};
}
/** Best display filename for an output, falling back to the work product title. */
export function outputFilename(item: IssueOutputItem): string {
return item.metadata?.originalFilename || item.title || "output";
}
export function getPromotedOutputAttachmentIds(workProducts: IssueWorkProduct[] | null | undefined): Set<string> {
return new Set(
getIssueOutputs(workProducts).items.flatMap((item) => {
const attachmentId = item.metadata?.attachmentId;
return attachmentId ? [attachmentId] : [];
}),
);
}