fix(skills): preserve referenced files in large repository imports (#16308)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use skills imported from project repositories.
> - Large repositories fall back to isolated skill folders.
> - That fallback drops shared files and turns advisory links into
import failures.
> - This pull request retains bounded referenced files and keeps missing
references advisory.
> - Users can import valid skills such as test-audit without editing the
source repository.

## Linked Issues or Issue Description

Refs #16066. Refs #16256. Refs #16288. Related draft #15730 adds
optional reference selection to individual skill packages. This fix
applies automatic repository fallback and does not add that selection
UI.

**What happened?**

A project source showed 86 warnings. It rejected 23 skills solely
because references were absent from the isolated package. It also
treated web routes and inline Markdown examples as files. For
test-audit, repository scripts were resolved inside the skill folder.

**Expected behavior**

Retain referenced files that fit the package limits. Missing references
must not block an otherwise valid selected skill. Preserve content
validation and installed shared files.

**Steps to reproduce**

Import paperclipai/paperclip at commit
09a2f0b7b0 as a repository source. Select
test-audit. Inspect its references and saved files. Repeat with a skill
whose optional reference is missing.

## What Changed

- Add a bounded fallback that keeps skill folders and transitive
Markdown references with their original repository paths.
- Resolve existing repository-root resource paths. Ignore site-relative
web links and inline Markdown examples.
- Keep missing references advisory, including explicit imports and
automatic project imports.
- Share fallback with file previews. Reinspect cached dependency
failures at an unchanged upstream commit.
- Retain existing versions when fallback cannot preserve installed
shared files. Keep file limits, content audits, authorization and work
budgets.
- Update product contracts and Skills maps. No schema or public API
changes.

## Verification

- Red/green: three original regression cases failed before the fix and
passed afterward. Two review controls also reproduced omitted directory
siblings and duplicate blob reads before their fixes.
- Focused scanner, repository package, persistence, project
synchronization and GitHub access tests: 126 cases passed across local
and isolated Linux runs.
- Full recursive typecheck and `pnpm build` passed locally on
`555bd3fc90314dbf23ed0ffe7034429946c1a471`. The [full Paperclip CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/38596749890)
passed on that head, including all general, serialized, Runner and eight
browser test shards. The duplicate local `pnpm test:run` was stopped
after CI completed (exit 143); it is not claimed as a completed local
full-suite pass.
- Real GitHub scan at the pinned public commit: 77 skills, 454 retained
files, 9,117,635 bytes. test-audit has no missing references. The full
source reports 23 warnings, including remaining file validation and
genuine missing-reference notices.
- Local browser: scanned the public repository, selected test-audit and
create-issue-interaction-ui, and imported both. The second skill
retained an advisory missing-reference warning. Both remained installed
after reload. The saved package exposes `scripts/run-vitest-stable.mjs`
(25,813 bytes) and `scripts/general-server-shard.mjs` (2,813 bytes). No
GitHub responses were mocked. No model execution was needed for this
import fix.
- Map checks and Git whitespace checks passed. Greptile is 5/5 on the
current head with both review findings resolved. The external Superagent
security scan concluded neutral/inconclusive after its agent exceeded
the prompt deadline. A check-run retry request returned HTTP 404. All
Paperclip CI gates passed, but merge is held pending a conclusive
security scan; this is not an all-checks-green claim.

## Risks

- Dependency detection covers explicit Markdown links and resource
paths. It does not infer all script imports or runtime dependencies.
- Referenced content can exceed package limits. The existing bounds
still apply, and warnings disclose omissions.
- Content validation still rejects invalid entrypoints and omits
rejected support files. This change does not make every upstream skill
importable.
- No migration is required. Existing sources with the old dependency
error are inspected again on refresh.

## Model Used

OpenAI Codex, GPT-6 family. The runtime did not expose the exact serving
model ID or context window. Used reasoning, code editing, terminal
execution and browser tools. No subagents were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/github-skill-source.ts

@@ -6,6 +6,7 @@ import { notFound, unprocessable } from '../errors.js';
 import { assertSkillSnapshotPath, snapshotFile, skillFileBytes } from './skill-snapshot.js';
 import { indexSkillPackagePaths, inspectSkillPackage } from './skill-package-inspection.js';
 import { auditSkillSnapshot, classifyInventoryKind } from './company-skills.js';
+import { skillRepositoryDependencies } from './skill-repository-dependencies.js';
 
 import type { GitSkillSnapshot, GitSkillSnapshotOptions, GitSkillTreeEntry as TreeEntry } from './skill-source-git-snapshot.js';
 
@@ -17,7 +18,7 @@ export interface GitHubRead {
   readonly grantId?: string | null;
 }
 export type DiscoveredSkill = SkillSourceCandidate & { files: CompanySkillVersionFileInventoryEntry[] };
-export type ScannedSkillSource = SkillSourceDiscovery & { skills: DiscoveredSkill[]; defaultBranch: string; treeSha?: string; downloadBytes?: number; scanBytes?: number; repositoryFiles?: CompanySkillVersionFileInventoryEntry[] };
+export type ScannedSkillSource = SkillSourceDiscovery & { skills: DiscoveredSkill[]; defaultBranch: string; treeSha?: string; downloadBytes?: number; scanBytes?: number; repositoryFiles?: CompanySkillVersionFileInventoryEntry[]; repositorySubset?: boolean };
 const MAX_FILE_BYTES = 1024 * 1024;
 const MAX_SCAN_BYTES = 100 * 1024 * 1024;
 const MAX_SCAN_PACKAGES = 1_000;
@@ -34,6 +35,7 @@ export interface SkillScanOptions {
   onProgress?: (event: SkillSourceScanUpdate) => void | Promise<void>;
   /** Discovery needs audited manifests, not the full contents retained for import. */
   retainFiles?: boolean;
+  repositorySubset?: boolean;
 }
 
 type SkillScanInput = { packageMode?: 'skills' | 'repository'; repositoryUrl: string; trackingRef?: string; commitSha?: string; onlySkillPath?: string };
@@ -49,20 +51,15 @@ export async function scanGitHubSkillsWithPackageFallback(input: SkillScanInput,
   catch (error) {
     if (input.packageMode !== 'repository' || !(error instanceof Error) || !/^Repository package exceeds/.test(error.message)) throw error;
     options.signal?.throwIfAborted();
-    const scan = await scanGitHubSkills({ ...input, packageMode: 'skills' }, providerRead, options);
-    rejectIncompleteSkillPackages(scan);
-    scan.warnings.push('Only skill packages were imported because the repository exceeds the supported package size.');
-    return scan;
-  }
-}
-
-export function rejectIncompleteSkillPackages(scan: ScannedSkillSource): void {
-  for (const skill of scan.skills) {
-    if (!skill.error && skill.inspection?.references.length) {
-      skill.error = 'This skill depends on files that its imported package cannot provide. Its dependencies could not be preserved. Import a supported repository package or include the dependencies in this skill package.';
+    try {
+      return await scanGitHubSkills(input, providerRead, { ...options, repositorySubset: true });
+    } catch (subsetError) {
+      if (!(subsetError instanceof Error) || !/^Repository package exceeds/.test(subsetError.message)) throw subsetError;
+      const scan = await scanGitHubSkills({ ...input, packageMode: 'skills' }, providerRead, options);
+      scan.warnings.push('Only skill folders were included because the repository and its referenced files exceed the supported package size. Missing references are listed below.');
+      return scan;
     }
   }
-  scan.candidates = scan.skills.map(({ files: _files, ...candidate }) => candidate);
 }
 
 async function scanRepository(input: SkillScanInput, providerRead: GitHubRead, options: SkillScanOptions): Promise<ScannedSkillSource> {
@@ -151,12 +148,28 @@ async function scanRepository(input: SkillScanInput, providerRead: GitHubRead, o
       if (declaredBytes > MAX_SCAN_BYTES) throw unprocessable('Skill packages exceed the 100 MB scan limit.');
     }
     const warnings = entries.filter(e => e.mode === '120000' || e.type === 'commit').map(e => `${e.path} — ${e.mode === '120000' ? 'Symlink' : 'Submodule'} couldn’t be included.`);
-    const repositoryInventory = input.packageMode === 'repository' ? entries.filter(entry => entry.type !== 'tree') : null;
+    const dependencyBlobs = new Map<string, Buffer>();
+    let dependencyBytes = 0;
+    const repositoryInventory = input.packageMode !== 'repository' ? null : options.repositorySubset
+      ? await skillRepositoryDependencies(entries, [...inventories.values()].flat(), selectedDirectories, async entry => {
+        if ((entry.size ?? 0) > MAX_FILE_BYTES) return null;
+        options.signal?.throwIfAborted();
+        const bytes = dependencyBlobs.get(entry.sha) ?? await snapshot.readBlob(entry, options.signal);
+        if (bytes.length > MAX_FILE_BYTES) return null;
+        dependencyBytes += bytes.length;
+        if (dependencyBytes > MAX_REPOSITORY_BYTES) throw unprocessable('Repository package exceeds the 61 MB runtime limit.');
+        dependencyBlobs.set(entry.sha, bytes);
+        return bytes;
+      }) : entries.filter(entry => entry.type !== 'tree');
     if (repositoryInventory && repositoryInventory.length > MAX_REPOSITORY_FILES) throw unprocessable('Repository package exceeds the 9,990 file limit.');
     const readable = repositoryInventory?.filter(entry => entry.type === 'blob' && ['100644', '100755'].includes(entry.mode) && (entry.size ?? 0) <= MAX_FILE_BYTES);
     if (readable && readable.reduce((sum, entry) => sum + (entry.size ?? 0), 0) > MAX_REPOSITORY_BYTES) throw unprocessable('Repository package exceeds the 61 MB runtime limit.');
     // A single bounded batch supplies both skill manifests and support files.
-    const blobs = readable && snapshot.readBlobs ? await snapshot.readBlobs(readable, options.signal) : new Map<string, Buffer>();
+    const blobs = dependencyBlobs;
+    if (readable && snapshot.readBlobs) {
+      const remaining = readable.filter(entry => !blobs.has(entry.sha));
+      if (remaining.length) for (const [sha, bytes] of await snapshot.readBlobs(remaining, options.signal)) blobs.set(sha, bytes);
+    }
     let totalBytes = 0;
     const readBlob = async (entry: TreeEntry) => {
       if ((entry.size ?? 0) > MAX_FILE_BYTES) return null;
@@ -217,6 +230,7 @@ async function scanRepository(input: SkillScanInput, providerRead: GitHubRead, o
       const inventory = repositoryInventory!;
       repositoryFiles = [];
       const packageWarnings = [...warnings];
+      if (options.repositorySubset) packageWarnings.push('Only skill folders and referenced files were included because the repository exceeds the supported package size.');
       totalBytes = 0;
       Object.assign(progress, { currentPath: null, checkedFiles: 0, totalFiles: inventory.length });
       for (const entry of inventory) {
@@ -272,7 +286,7 @@ async function scanRepository(input: SkillScanInput, providerRead: GitHubRead, o
         if (skill.inspection) skill.inspection.references = skill.inspection.references.filter(ref => !included.has(ref.resolvedPath));
       }
     }
-    return { treeSha: snapshot.treeSha, downloadBytes: snapshot.downloadBytes,
+    return { treeSha: snapshot.treeSha, downloadBytes: snapshot.downloadBytes, ...(options.repositorySubset ? { repositorySubset: true } : {}),
       scanBytes: Math.min(MAX_SCAN_BYTES, snapshot.entries.reduce((sum, entry) => sum + (entry.type === 'blob' ? entry.size ?? MAX_SCAN_BYTES : 0), 0)),
       ...(input.packageMode ? { packageMode: input.packageMode } : {}), ...(repositoryPackage ? { repositoryPackage } : {}),
       ...(repositoryFiles && options.retainFiles !== false ? { repositoryFiles } : {}),
@@ -283,7 +297,7 @@ async function scanRepository(input: SkillScanInput, providerRead: GitHubRead, o
 
 /** Reauthorize the caller and re-audit the selected package; never trust a client-supplied manifest. */
 export async function previewGitHubSkillFile(input: SkillSourcePreviewRequest, read: GitHubRead): Promise<SkillSourceFilePreview> {
-  const scan = await scanGitHubSkills({ ...input, onlySkillPath: input.skillPath }, read);
+  const scan = await scanGitHubSkillsWithPackageFallback({ ...input, onlySkillPath: input.skillPath }, read);
   if (scan.commitSha.toLowerCase() !== input.commitSha.toLowerCase()) throw unprocessable('The preview commit did not match the requested snapshot.');
   const skill = scan.skills.find(candidate => candidate.path === input.skillPath);
   if (!skill) throw notFound('Skill package not found at this commit.');