mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 21:07:40 +01:00
DottaandPaperclip
c08da3e0a7
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The skill library lists current company skills and their agent attachments. > - Each list request also reconciles bundled skills, folders, and local files. > - These reads return full agent records and large metadata that their callers do not use. > - Conditional requests do this work before they return an unchanged response. > - This pull request selects the fields each caller needs and avoids unused folder lookups. > - The result is less database transfer and fewer round trips while reconciliation stays current. ## Linked Issues or Issue Description Refs #17941. That change reuses bundled categories within an import. This change removes other repeated work in the same list path. **What happened?** Warm skill-list requests still made 119 SQL calls in a local HTTP fixture. The folder pass returned full skill metadata. The local-source pass also returned remote skills. Attachment counts loaded full agent configurations and monthly spend. Bundled reconciliation resolved folder paths that its release check did not use. **Expected behavior** The list must preserve its current result, attachment counts, access rules, local-file reconciliation, and bundled release checks. Internal reads should return only the data their callers need. **Steps to reproduce** 1. Use a company with many remote skills and agents with large saved configurations. 2. Read its skill list, then repeat the read with its ETag. 3. Measure the queries and response values before the server returns `200` or `304`. 4. Run `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-skill-list.ts --samples=5 --network-delay-ms=3.5` for a synthetic reproduction. **Paperclip version or commit** Baseline service: `e1ba77f06771a27114b3999004b0ca5e3c172dd4`. The branch is based on current master. The paired mode changes only the skill-service module; both versions use the same imported helpers. **Deployment mode** Built from source. Local HTTP server and disposable PostgreSQL 17. No hosted deployment is part of this PR's local proof. ## What Changed - Read only the desired-skill selections of nonterminated company agents for attachment counts. Resolve each agent's references once against the full visible skill list. Preserve IDs, keys, aliases, duplicate handling, and ambiguous slugs. - Read only `sourceKind` from metadata for bundled folder classification. - Limit local-source reconciliation to local-path rows. - Skip folder-path enrichment only for bundled reconciliation. Normal imports and list responses keep their folder paths. - Add real-database byte regressions and a local HTTP benchmark. The benchmark can alternate the old and new services against one database, actor, and process. It checks complete response equality and conditional responses. The company-skill service owns these reads. The query predicates and freshness checks remain at their existing owners. This change adds no cache, schema change, or new background work. ## Verification - The attachment byte regression failed before the fix: 601,418 captured response-value bytes exceeded its 2,000-byte bound. The same case passed after the fix. It checks fresh configuration edits, IDs and keys, duplicate aliases, ambiguous slugs, paused agents, terminated agents, and company scope. - 111 focused service and authorization tests passed against isolated real PostgreSQL 17 databases. They include full result equality, retained stored Markdown and metadata, local-file updates, release checks, project visibility, and import authorization. - The default embedded PostgreSQL probe skipped the first local attempt on this Mac. The completed run used a local-only bootstrap adapter for disposable databases. Queries, migrations, services, authorization, and filesystem work used their real implementations. The adapter is not part of this PR. - In repeated synthetic HTTP byte runs, calls fell from 119 to 98 per request. Result-value bytes fell from about 7.75 MB to 1.64 MB per request, about 79%. These values are measured before Drizzle decoding. They exclude protocol framing and do not measure provider billing. - A separate-process run without the byte observer did not show a latency gain. The paired experiment removes that comparison gap: it alternates both services against the same database and checks identical complete responses. On final commit `3076fe1402085f7286ff58e642d7caca892a5f80`, five observer-free pairs had median times of 2.582 s before and 1.850 s after (28.3% lower). The candidate was faster in all five pairs. Its range was 1.802–2.199 s; the baseline range was 2.327–2.725 s. Every normal response had the same complete body hash, and both versions returned the expected unchanged conditional responses. These are local service/HTTP timings with synthetic data and 3.5 ms delay per network direction. They are not hosted acceptance. - Final-head workspace typecheck and build passed. The full local test command did not complete because embedded PostgreSQL bootstrap failed on this Mac; I stopped that owned runner after 16 native-runtime cases failed. All 16 cases then passed when rerun alone against disposable PostgreSQL 17 using their existing external-database path. The 111 focused PostgreSQL 17 tests above also completed successfully. This does not constitute a completed full local suite; all current-head remote CI gates passed (55 successful checks and 2 skipped). Module-boundary and whitespace checks pass. An initial nested package command used the global pnpm; subsequent commands were pinned to pnpm 9.15.4. - The first canary dry run failed while installing an unrelated standalone package: npm returned 404 for the newly published Rollup 4.66.0 tarball. The exact download subsequently returned HTTP 200. The one retry passed without code changes. - Benchmark commands: add `--capture-values=false` for observer-free timings. Add `--baseline-ref=e1ba77f06771a27114b3999004b0ca5e3c172dd4 --capture-values=false` for paired service acceptance. The normal mode uses the production router. The paired mode uses equivalent default-list HTTP handlers and a synthetic authenticated board actor; it does not test login. ## Risks The SQL selection must stay in sync if the desired-skill format or folder classifier needs new fields. Existing parsing and reference resolution still interpret the selected data. The synthetic fixture has large unused payloads. It proves the mechanism and preserves required content. Its percentage is not a projection of whole-instance transfer or billed savings. Hosted latency still needs verification after merge and deployment. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell execution, real PostgreSQL, and local HTTP testing. The host does not expose a precise backend revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>