fix(skills): preserve referenced files in large repository imports (#16308)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use skills imported from project repositories.
> - Large repositories fall back to isolated skill folders.
> - That fallback drops shared files and turns advisory links into
import failures.
> - This pull request retains bounded referenced files and keeps missing
references advisory.
> - Users can import valid skills such as test-audit without editing the
source repository.

## Linked Issues or Issue Description

Refs #16066. Refs #16256. Refs #16288. Related draft #15730 adds
optional reference selection to individual skill packages. This fix
applies automatic repository fallback and does not add that selection
UI.

**What happened?**

A project source showed 86 warnings. It rejected 23 skills solely
because references were absent from the isolated package. It also
treated web routes and inline Markdown examples as files. For
test-audit, repository scripts were resolved inside the skill folder.

**Expected behavior**

Retain referenced files that fit the package limits. Missing references
must not block an otherwise valid selected skill. Preserve content
validation and installed shared files.

**Steps to reproduce**

Import paperclipai/paperclip at commit
09a2f0b7b0 as a repository source. Select
test-audit. Inspect its references and saved files. Repeat with a skill
whose optional reference is missing.

## What Changed

- Add a bounded fallback that keeps skill folders and transitive
Markdown references with their original repository paths.
- Resolve existing repository-root resource paths. Ignore site-relative
web links and inline Markdown examples.
- Keep missing references advisory, including explicit imports and
automatic project imports.
- Share fallback with file previews. Reinspect cached dependency
failures at an unchanged upstream commit.
- Retain existing versions when fallback cannot preserve installed
shared files. Keep file limits, content audits, authorization and work
budgets.
- Update product contracts and Skills maps. No schema or public API
changes.

## Verification

- Red/green: three original regression cases failed before the fix and
passed afterward. Two review controls also reproduced omitted directory
siblings and duplicate blob reads before their fixes.
- Focused scanner, repository package, persistence, project
synchronization and GitHub access tests: 126 cases passed across local
and isolated Linux runs.
- Full recursive typecheck and `pnpm build` passed locally on
`555bd3fc90314dbf23ed0ffe7034429946c1a471`. The [full Paperclip CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/38596749890)
passed on that head, including all general, serialized, Runner and eight
browser test shards. The duplicate local `pnpm test:run` was stopped
after CI completed (exit 143); it is not claimed as a completed local
full-suite pass.
- Real GitHub scan at the pinned public commit: 77 skills, 454 retained
files, 9,117,635 bytes. test-audit has no missing references. The full
source reports 23 warnings, including remaining file validation and
genuine missing-reference notices.
- Local browser: scanned the public repository, selected test-audit and
create-issue-interaction-ui, and imported both. The second skill
retained an advisory missing-reference warning. Both remained installed
after reload. The saved package exposes `scripts/run-vitest-stable.mjs`
(25,813 bytes) and `scripts/general-server-shard.mjs` (2,813 bytes). No
GitHub responses were mocked. No model execution was needed for this
import fix.
- Map checks and Git whitespace checks passed. Greptile is 5/5 on the
current head with both review findings resolved. The external Superagent
security scan concluded neutral/inconclusive after its agent exceeded
the prompt deadline. A check-run retry request returned HTTP 404. All
Paperclip CI gates passed, but merge is held pending a conclusive
security scan; this is not an all-checks-green claim.

## Risks

- Dependency detection covers explicit Markdown links and resource
paths. It does not infer all script imports or runtime dependencies.
- Referenced content can exceed package limits. The existing bounds
still apply, and warnings disclose omissions.
- Content validation still rejects invalid entrypoints and omits
rejected support files. This change does not make every upstream skill
importable.
- No migration is required. Existing sources with the old dependency
error are inspected again on refresh.

## Model Used

OpenAI Codex, GPT-6 family. The runtime did not expose the exact serving
model ID or context window. Used reasoning, code editing, terminal
execution and browser tools. No subagents were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/PRODUCT.md

@@ -111,10 +111,12 @@ default branch marked. People can also enter a branch name that is not listed.
 Saving checks GitHub access and the selected ref before changing tracking; failed
 saves keep the selection and installed content. Project refs remain owned by project
 configuration so changing a source cannot redirect another project's tracking.
-If a tracked repository grows beyond the supported package size, its installed
-repository skills and shared dependencies remain available until a safe refresh
-can succeed. Initial oversized imports can use self-contained skill packages;
-packages with omitted dependencies are invalid and are not installed.
+If a repository exceeds the supported package size, Paperclip first packages its
+skill folders and referenced repository files, preserving their original paths.
+Missing references are advisory: a valid selected skill can still be imported.
+Unavailable files are listed, and existing installed dependencies are retained
+when a bounded refresh cannot preserve them. File limits and content validation
+still apply; importing never executes repository code.
 
 Project-derived skills inherit current project access. Access through any originating
 project is sufficient, including the responsible-user restrictions on agent execution.

doc/SPEC-implementation.md

@@ -2110,11 +2110,15 @@ unavailable. Preserve current ownership and newer-work fences. See
   be paused independently of selection. Project sources select future additions unless
   excluded; existing manually imported sources retain their selections. Legacy projects
   adopt through an explicit company-scoped, project-authorized endpoint, never a GET.
-- Repository size fallback must not discard installed shared dependencies or
-  publish packages with detected omitted dependencies. Retain affected versions
-  and report the size problem. Self-contained packages can still update through
-  fallback. Project sources retry complete repository packaging and return to it
-  when supported; selections and exclusions remain unchanged.
+- Repository size fallback first retains skill folders and transitively referenced
+  repository files within the existing file/byte limits, preserving original paths.
+  If that set is still too large, bounded skill folders remain importable. Missing
+  references are advisory and do not invalidate otherwise valid selected skills.
+  Web routes and inline Markdown examples are not package dependencies. Preserve
+  installed versions if fallback would omit their previously installed shared files.
+  Project sources retry complete packaging; selections and exclusions remain unchanged.
+  Refresh reinspects cached dependency-rejection errors once even at an unchanged
+  upstream commit. Content validation and company/project access remain enforced.
 - Unattended refresh stores setup identity and connection/grant identifiers, never token
   values. Current membership, connection authorization and skill policy are revalidated.
   Unrecoverable legacy setup identities require explicit connection repair. Retries are

doc/SPEC.md

@@ -722,6 +722,9 @@ the runtime. A declarative manifest can suggest public entrypoints and setup req
 new imports keep repository files together and select all discovered skills. Missing
 references, setup problems and omitted files produce persistent warnings. Files rejected
 for unsafe execution or secret disclosure are excluded without blocking other files.
+Oversized repositories use a bounded package of skill folders and referenced files
+with their original paths. Missing references do not block a valid selected skill;
+fallback preserves previously installed shared dependencies or retains the old version.
 The installed file viewer exposes saved repository dependencies. Importing never runs
 setup or configures host tools. New
 upstream skills require reviewed selection; removed or deselected skills remain