mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 17:33:51 +02:00
ea4aa185b0f3bc30be1c5fc160cdfd0eb433fe41
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
df4ee06e2f |
feat(skills): select and validate tracked GitHub branches (#16288)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Skills can track a branch in an authorized GitHub repository. > - Sources previously offered only a default-branch repair action. > - People need to select a listed branch or enter a branch that GitHub has not listed. > - The server must validate the selected ref with the caller's authorized connection. > - This pull request connects the approved branch picker to production APIs. > - Installed skills remain usable while the source changes its tracked ref. ## Linked Issues or Issue Description **Subsystem affected** Skills UI, shared API contracts, GitHub source services, and agent API discovery. **Problem or motivation** The default-branch repair action does not let people choose another branch. The settings service also validates the old ref when a caller supplies a new ref other than HEAD. A missing tracked branch can prevent connection repair even when the selected account has repository access. **Proposed solution** Load a bounded branch list with the authorized connection. Mark the default branch. Search listed branches or accept a typed ref. Validate the selected ref before saving. Keep the chosen value after a load error, invalid ref, or concurrent update. Save the selected connection and branch together during connection repair. **Alternatives considered** A default-branch button does not support a named branch. A text field alone does not help people find the repository default or available branches. **Roadmap alignment** This improves the existing Skills source manager. It does not add a new skill import system. **Additional context** Refs #16066 and #16256. This is the production implementation of the approved branch-selection Storybook journeys. ## What Changed - Add an authorized branch-list endpoint and shared response/query types. Register it in agent API discovery. - Validate the requested ref, including the returned commit SHA. Preserve source leases, revision checks, project ownership, skill identities, selections, and installed versions. - Add the searchable branch picker to Sources and connection repair. Allow typed refs during branch discovery and after discovery fails. - Keep save errors in the dialog. Retain the selected branch through a concurrent source update. - Use the production components and API client in Storybook. Add keyboard, touch, loading, retry, invalid-ref, connection-repair, and concurrent-update browser journeys. - Update product documentation and the feature/access maps. ## Verification - Red/green: two new database regressions fail against the previous service and pass against the implementation. - Linux database/service suite: 85 tests passed. This includes source visibility, human/agent authorization, selected connections, immutable versions, and invalid refs. - Rebased branch and review fixes: 94 focused UI, route, branch-validation, and fixture tests passed. Two conflict-message regressions and the stale-retry fixture regression fail before the fix and pass after it. - Storybook build passed. All 26 focused browser journeys passed. - Repository typecheck and build passed. Final UI typecheck passed after the review fixes. - The full local Vitest run was interrupted when the checkout and its dependencies disappeared. It reported missing working-directory/Postgres files and worker startup errors. Its two timed-out tests pass after the checkout was restored. The complete CI suite passed on the final head, `cf5d4c8f0a`: [CI run](https://github.com/paperclipai/paperclip/actions/runs/38571068332). One unchanged workspace timestamp test required a single rerun. Its release-minus-one-millisecond value can equal the original confirmation time. The rerun passed. - Live acceptance passed on implementation head `5dd935bf14` in an isolated macOS arm64 instance with Node 24.21.0 and an authorized private GitHub repository. The UI loaded the actual default branch, saved branch and selected connection together, and preserved the installed skill/version IDs after reload. An invalid typed ref retained the draft and showed the provider error. A valid typed HEAD ref then saved successfully. The owned runtime and database were stopped after the test. Focused unit, Linux database, and browser suites passed again after the error-handling review fixes. - Greptile reviewed final head `cf5d4c8f0a` at 5/5. Both review threads are resolved; the fresh review has no remaining findings. - Token gates and map/catalog checks passed. - Review entry points: `Skills / Project repository skills / 03 Branch selection components` and `04 Branch selection journeys`. Run `pnpm storybook` to inspect them. - Browser command: `pnpm exec playwright test --config tests/storybook-visual/playwright.config.ts tests/storybook-visual/skill-source-branch-selection.spec.ts tests/storybook-visual/project-repository-skills.spec.ts`. - Manual check: open an independent GitHub source, choose a new branch, select the default or type a ref, save, and reload. Try an invalid ref and confirm the selection remains in the dialog. For a project-bound source, change its ref in project settings. ## Risks - GitHub discovery is bounded to 100 listed branches. The default branch is always included. Typed refs remain available. - A source can change while the dialog is open. The API rejects a stale revision and the UI keeps the draft for retry. - A ref change schedules the existing refresh pipeline. Installed versions remain available until a successful refresh imports new content. - No database migration or credential storage change is required. Existing HEAD requests remain supported. ## Model Used OpenAI Codex, GPT-6 family. The serving model's exact identifier and context-window size are not exposed in this session. Used reasoning, repository tools, code execution, and browser tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
473c80d13f |
Automatically import and update project repository skills (#16066)
Automatically import skills from new GitHub project repositories and update enabled sources in the background. Preserve project-derived access, saved selections, immutable execution versions and explicit pins; add legacy adoption, refresh and connection repair in Sources. Include durable scheduling and bounded work credits, production Storybook journeys, real GitHub-to-routine acceptance receipts and final-head CI/security/Greptile qualification. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2ad36f4ff2 |
Fix skill source removal with a keep-or-delete choice (#16150)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub sources import skills and keep their updates together.
> - Disconnect only disabled a source and kept its read-only imports.
> - A disabled source could not be removed or imported again.
> - Removing a source needs an explicit decision about its imported
skills.
> - This pull request removes the source after the user keeps editable
copies or deletes the imports.
> - Existing disabled sources use the same removal flow.
## Linked Issues or Issue Description
Refs #15731, #14713.
**What happened?**
Disconnect left the repository on Sources. Its Disconnect action was
then disabled. The saved repository still blocked a new import.
**Expected behavior**
Remove the repository from Sources. Require a choice to keep editable
copies or delete its imports. Allow removal of sources that are already
disconnected.
**Steps to reproduce**
1. Import a repository in Skills → Sources.
2. Choose Disconnect source.
3. Observe the disconnected row and disabled removal action.
4. Try to import the repository again.
**Paperclip version or commit**
|
||
|
DottaandPaperclip
|
d432dc7fa3 |
Add GitHub-synced skill sources (#14713)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Company skills supply instructions and files to those agents. > - GitHub imports already exist, but users cannot manage repositories as skill sources. > - Repository refresh also needs caller-authorized access and complete local packages. > - This pull request adds Sources inside Skills and reuses GitHub connections from Apps. > - Installed snapshots let agents use skills without fetching GitHub during a run. > - Manual refresh preserves skill identity and leaves failed imports on their last good version. ## Linked Issues or Issue Description **Subsystem affected** Cross-cutting: skills UI, server, database, shared contracts, and runtime materialization. **Problem or motivation** Users keep skills in GitHub repositories. They need a clear way to select, import, and refresh those skills. Existing imports do not expose repository management or consistently preserve supporting files. **Proposed solution** Add company-scoped skill sources. Browse repositories from all accessible GitHub connections, or paste a public repository or branch URL. Select whole skill packages, inspect included files and reference warnings, and install complete, immutable snapshots. Refresh each source manually. **Alternatives considered** Project repository settings hide the workflow from Skills. A second GitHub connector would duplicate credentials and grants. Upstream editing and PR creation are separate work. **Roadmap alignment** This implements the Skills Manager direction in ROADMAP.md. The maintainer requested this scope and reviewed the component and full-app journey stories before implementation. Related reports: Refs #10285, Refs #10949, Refs #13464. Related work: #14356, #13656, #9268. ## What Changed - Add source and entry records, an idempotent migration, company-scoped APIs, and legacy GitHub import adoption. - Reuse current caller grants and credential refresh. Combine and deduplicate repository inventories across accessible connections. Pasted public URLs also prefer the active user’s authorized connections. Tokens stay in the Git child environment, never argv or disk. - Fetch a shallow Git snapshot at one immutable commit. Scan the full local tree, including hidden and nested folders. Read Git objects without checkout or archive transformations and enforce nested package boundaries. - Bound Git downloads to 128 MiB and three minutes. Cancel active process groups and remove incomplete downloads. Preserve cancellation and deadlines while progress drains; close stalled HTTP progress streams after 30 seconds. Reuse caller-scoped temporary snapshots for preview/import after reauthorization. - Index repository package boundaries once and cap expanded work at 1,000 packages, 10,000 files, and 100 MiB, including repeated copies of shared blobs. Bound path depth and the shared path index. Discovery keeps audited manifests without retaining all package bodies. - Resolve moving branches before fetching so unchanged discovery reuses caller-scoped snapshots. Limit active scans, scan frequency, and new downloads per caller and company; quotas apply before metadata reads and across connections, and cached scans do not consume the download quota. - Store complete versions with script content, binary bytes, and executable modes. Preserve these through copies, runtime caches, and runner packaging. - Stage downloads before publication. Use source leases, revision checks, and transactional activity records. Keep installed versions after failures, upstream deletion, deselection, and disconnect. - Add the approved import flow, Sources page, selection tree, provenance, read-only Studio behavior, and saved return from GitHub setup. - Add package manifests, commit-pinned file previews, and separate runtime requirements and reference warnings. Supporting files are included together; nested skills remain independently selectable. Preview requests reauthorize the caller and re-audit package content. - Show installed skills as compact links beneath each source. Repository titles open GitHub. Keep Refresh, Select skills, and Disconnect source in a three-dot menu. Source rows omit the branch, imported count, and refresh timestamp; action alignment and repository titles work at narrow widths. - Stream discovery metadata over an opt-in NDJSON response. Show measured Git download progress and real package/file counts, animate newly checked skills, support cancellation, and require a complete scan before selection. Keep the existing JSON API. - Retain component stories and add a separate full-app journey story group. Include fixed progress states and interactive scan, large-repository, interruption, and saving stories. - Update Skills documentation and product contracts. Suppress private GitHub skill references in telemetry. Privacy review requested for the telemetry changes. ## Verification - Local repository typecheck, full build, token gates, and Storybook build passed during this work. Focused transport, authorization, scanner, persistence, route, and UI tests pass. The final UI refinement passes all eight focused UI tests, UI typecheck/build, and token gates. The scanner resource and repeated-discovery fixes pass 132 focused scanner, transport, authorization, source-service, route, and rate-limit tests, plus server typecheck/build. Full-suite verification comes from CI; the older full local Vitest run was stopped after unrelated chat failures and a font-test failure, all of which passed in fresh focused runs. At commit `1098d5996`, all 54 active checks pass; two optional Storybook jobs are skipped. CI covers repository typecheck, build, the full test suites, browser shards, and the canary dry run. Greptile is 5/5 with no open findings; the security scan also passes. - Adversarial scanner tests verify repeated-blob byte accounting with and without declared sizes, package/file/path caps, one-time repository indexing, metadata-only discovery audits, and nested package boundaries. Additional tests cover branch movement, snapshot reuse, caller/company quotas, isolation across connections, active-lease cleanup, quota recovery, and rejection before any metadata API call. - Real Git tests verify hidden paths, exact binary bytes, executable modes, export-ignore preservation, symlink/submodule reporting, pinned commits, caller-scoped cache reuse, cancellation, cleanup, and credential isolation. Regression tests hold both download slots with permanently blocked progress callbacks, verify timeout/cancellation cleanup and retry, and exercise HTTP backpressure cancellation. Access tests cover automatic public-URL connection selection and revoked grants. Database tests verify company and grant audiences. - Live isolated browser test: the public `anthropics/skills` scan now completes and discovers all 20 skills without connecting an account. Imported canvas-design with all 83 files, opened it from Sources, and verified the installed binary-font preview/download control. Package previews also expose the complete file inventory before import. Cancelled an active Git download and retried successfully to all 20 discovered skills; the browser displayed measured download progress. The current audits reject four other packages; eligible selections remain importable. - Browser checks verify the simplified source rows at desktop and narrow widths, keyboard navigation into the actions menu, Refresh from the menu, selection, and fixture disconnect with installed skills retained. Storybook includes a menu-open checkpoint and a 320px layout. - Storybook includes receiving/preparing download checkpoints and a timed full-app import journey, plus cancellation, retry, large-repository, and saving states. Streaming tests cover split UTF-8 frames, incomplete streams, late responses, cross-company requests, HTTP errors, and JSON compatibility. - Earlier live acceptance on this PR imported `stitch-skill` with `DESIGN.md`, assigned it to an agent, disconnected its source, and ran a successful Studio test that read both installed files. An editable copy changed independently. Both Skills variants, mobile selection, and return from GitHub setup were exercised. - Private access, revoked credentials, OAuth success return, binary/script preservation, concurrent refresh, transaction rollback, version pins, and legacy adoption have automated coverage. A real private-repository OAuth grant was not created during this test. ## Risks - The migration groups recognizable legacy imports without provider calls. Their first successful refresh completes the local package snapshot. - Reference checks are advisory. They cover Markdown links and explicit relative resource paths, not arbitrary runtime dependency graphs. Preview text is capped at 64 KiB; imported bytes remain complete. - Git must be installed on the server. Shallow fetches still download the branch snapshot, including files outside selected packages. Downloads have size/time/concurrency limits. Temporary caches are bounded and caller-scoped. GitHub API quota still applies to repository metadata and the connection picker; content no longer uses per-file API requests. Failed scans retain installed content. - Sources depend on the current caller's GitHub access. A saved connection does not grant access to another person's token. - GitHub script support and immediate manual refresh are explicit maintainer-approved requirements. The operator trusts the selected repository and accepts upstream script and executable-mode changes on refresh. Static audits are not a sandbox or a guarantee of safe code; agents may later invoke installed helpers under their runtime permissions. Import and refresh do not execute scripts, hooks, package installation, or builds. Raw URL and skills.sh imports keep their prior script restrictions. - Source originals remain read-only. Refresh affects subsequent unpinned runs; explicit pins and active runs retain their versions. - The telemetry change removes source-managed GitHub identifiers from skill-reference events. It introduces no event or field. Please review the privacy boundary. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code execution, and browser tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |