From f1911a4c535687be12c2638f35a4f42872f8e16e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 29 Oct 2026 12:19:37 -0500 Subject: [PATCH] perf(skills): move repository packages out of Postgres (#18082) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205 ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents receive selected company skills before each turn. > - Imported skills can need files from the whole source repository. > - The current path stores those files in Postgres and reads large JSON payloads during preparation. > - This pull request stores verified packages in the configured file or object store and keeps preparation queries small. > - It also converts old rows automatically and restores missing files from the exact saved commit. > - The result reduces database traffic while preserving skill versions and repository dependencies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Company skill import, version storage, and preparation before agent turns. **Current behavior** Repository snapshot bodies live in JSONB. Skill preparation can read those bodies again even when the files already exist on disk. Historical packages and native assets also need a bounded cleanup path. **Proposed behavior** Store package archives and manifests through the existing storage service. Keep version identities, receipts, hashes, and recovery metadata in Postgres. Convert legacy rows after the stored package passes read-back verification. A missing package can use legacy bytes or its exact Git commit under current repository authorization. No separate data-migration command is required. **Reason and benefit** The warm path sends no repository bodies through SQL and reads no object bytes. Cleanup runs through the existing deadline coordinator. Current versions, active runs, explicit pins, and authored content retain their protection. **Breaking changes** The normal application schema upgrade is required. Single-version and skill-file responses keep their existing shape. Version lists, current-version summaries, and test-run version summaries now contain file names and kinds without file bodies. Diff and restore fetch selected versions through the existing single-version endpoint. Older server builds must not write after conversion unless they support the new package format. The storage override and rollback limits are documented in `doc/skill-package-storage.md`. **Additional context** Related work: #15731 preserves repository dependencies; #17923 changes version-cache scheduling; #18080 changes delivery to remote Codex homes. This change owns package persistence, acquisition, and expiry. It does not replace those delivery or scheduler changes. ## What Changed - Add company-scoped archive receipts, exact-commit recovery recipes, migration deadlines, and a durable object-deletion queue. - Add bounded archive upload, read-back verification, extraction, concurrent request coalescing, and verified local caches. - Convert old snapshots and versions automatically. Keep legacy bytes when verification fails. - Recover missing files through current source authorization and the original commit. Validate repository identity, file hashes, modes, and the aggregate hash. - Expire unused recoverable Git packages and native assets after 30 days. Preserve active pins and the durable copy of authored or unproven content. - Keep warm preparation queries metadata-only. Bound file digest caches and avoid repeated body hashing when file identities are unchanged. - Extend the existing storage contract with cancellation. Register package work with the existing coordinator. Optional conversion retries retain the original bytes and may wait for ingress without blocking sleep. - Add regression tests, operating guidance, a deep trace, benchmark receipts, and an optional maintenance command. ## Verification ### Final master integration and review repair — October 29, 2026 Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit `f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final two commits repair the Apex deletion findings below. - Keep both package-maintenance and version-retention notifications in their write transactions. - Keep the shared run-pin predicate and master's controller-lease deadline metadata. Package expiry reads the updated protected-version result. - Preserve every master migration entry. Generate migration **0415 (`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is byte-for-byte unchanged from the previously tested idempotent migration. Retain only the five newest schema snapshots. Migration numbering and safety checks pass. - Repair failed file/folder deletion: preserve a complete temporary subtree until transaction commit, restore on upload or SQL failure, and restore callback failures before releasing the mutation lock. Both PostgreSQL regressions reproduced missing files before the fix and now pass. They cover binary data, executable permissions, empty folders, unlisted files, unchanged database/version rows, successful retry after upload-claim expiry, and historical reads. - Reconcile lost commit replies under the same advisory and row locks. Check the saved version and source identity before restoring. Keep committed deletions intact and preserve backups when the database outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome tests pass; the prior code failed three of them. - Full workspace typecheck and production build pass on the final commit. All **204** focused PostgreSQL package/cache/retention/authoring and filesystem-boundary tests pass, with none skipped. - Apex review **18 passes 5/5**, with zero new comments on the exact final head. All **34** review conversations are resolved. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508) completes successfully on attempt 2. The head has **55 successful checks**, two neutral reports, two skipped checks, and a successful Snyk status. No checks are pending or failing. All eight browser shards pass. - Runner Vitest 2/2 timed out once in the unchanged ACPX controller-replacement fixture. The same job passed on the preceding commit. One unchanged CI retry passes all five ACPX recovery cases, **2,253** Runner package cases and **126** native server integration cases; their standard one and five skips remain. The initial timeout's root cause is not established. Local diagnosis also hit the fixture's eight-second session.open timeout with an authenticated Runner; no local Runner pass is claimed. - Both full local `pnpm test:run` attempts were stopped when Apex identified deletion defects and source repair became necessary; neither is claimed as a passing full local run. The final **204** focused tests, full workspace typecheck, and production build pass. Full-suite acceptance uses the final-head CI jobs. The superseded merge-head CI included runners terminated by shutdown signals; each source repair started a new full run. - Auto-merge is enabled for squash merge. The final head is conflict-free and all checks pass. GitHub's active master ruleset still requires a code-owner approval for the dependency files. The existing review requests are with `devinfoley`, `nickyleach`, and `forgottendev`; no policy bypass was used. - The earlier hosted stress and full native receipts below remain tied to `a2e597d800`; they are not relabeled as a deployment of this final integration. The earlier local PostgreSQL startup failure remains recorded below; PostgreSQL ran successfully for this integration verification. ### Full native skill execution — October 29, 2026 The full native-agent acceptance gap is closed on candidate `a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same disposable hosted TEST tenant as the stress test. A follow-up on the same task consumed a newly saved skill version. The task text asked for the assigned skill workflow. It did not contain the proof nonce or expected receipt. The assigned authored skill contained a Python verifier and a random package-local nonce. The agent read the actual installed skill files and executed that verifier against the imported, automatically converted 1,360-file repository package. It checked the repository file count and reference/binary bytes, and directly executed the repository's executable probe. Independent verification matched the emitted JSON to the nonce and expected byte hashes. The agent persisted the same receipt as its task response; native finalization committed, workspace finalization succeeded, and task state became Done. | Qualifying turn | End-to-end run time | Recorded skills.prepare | Repository files | Binary hex | Shell receipt SHA-256 | |---|---:|---:|---:|---|---| | Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` | `a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` | | Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 | `00ff07800a` | `dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` | Both successful shell receipts reported `probe: STRESS18082` and `assetSha256: e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their proof nonces, selected version IDs and receipt hashes differed, and both ran on the same task. These are observed timings, not latency SLOs; they include real provider work. The service stress-test timings remain a separate measurement boundary. Earlier failures remain in the evidence: - The original five-minute Daytona allocation timeout never reached model execution. After normal owned-stack restore, its durable cleanup record was `expired` / `success`. The cause of that original allocation timeout remains unknown; a later success establishes recovery only. - Initial continuation setup used unsupported generic agent-update fields and returned HTTP 409 before dispatch. Setup was corrected to use the supported skill-assignment and versioned instruction commands. - The intervening completed native turn read repository bytes and ran the probe, but omitted the verifier JSON under the older test agent's overly-specific instruction file. Its strict receipt validator failed; it is not counted as a qualifying turn. Correcting the test agent's instruction file and sending an explicit task-scoped wake produced the qualifying follow-up above. - The pinned application's immutable rate card omits `gpt-5.4-mini`, so completed usage triggered the normal block-on-unpriced budget policy. Supported append-only accounting corrections estimated the three completed runs using the [official OpenAI model rate card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75 input, $0.075 cached input, $4.50 output per million tokens, assuming standard processing. Estimates totaled 12.482655 cents. Original usage and missing-price records are preserved; the $1 hard stop and unpriced-usage guard stayed active. - Model shell attempts included a missing `xxd` command and incorrect guessed file paths. Those failed calls and subsequent successful actual verifier calls remain visible in complete event streams. No zero-tool-error claim is made. No product source changed during this continuation. The previously measured contention, cold-start and memory limits remain. The later merge resolution is covered by the separate integration checks above; these hosted receipts qualify their recorded candidate. ### Hosted stress test — October 29, 2026 **Hosted acceptance:** full native-agent execution and a changed-skill follow-up pass on the recorded hosted candidate. The final integration section above records the subsequent conflict resolution and its separate checks. Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this exact candidate sequentially on one disposable TEST tenant, with external Neon PostgreSQL and actual S3. The fixture contained **1,360 files / 22.4 MB / ten skills sharing one repository**. Measurements cover deployed skill-preparation and native-context services in an adjacent diagnostic process, not a complete agent turn. The container had **24 vCPU / 24 GB**. | Warm workload | Median before → after | p95 before → after | DB returned value bytes/preparation before → after | |---|---:|---:|---:| | One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 → 1,771 | | One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms | 22,610,389 → 1,771 | | Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 → 10,671 | | Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms | 224,739,030 → 10,671 | - Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding dedicated lock connections. DB numbers measure returned values before Drizzle decoding, not wire bytes. Baseline/candidate sample counts were 30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill samples are comparative evidence, not SLO estimates. - **Application-UID soak:** 800 ten-skill preparations over 10.3 minutes, four workers paced by 2.5 seconds; **zero failures**, median **559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726 accompanying health/company-list requests returned HTTP 200. Their end-to-end p95 was 569/502 ms, including network latency. The probe overlaps most of the soak. Soak DB-response capture was limited to its first 60 seconds. - **Important remaining costs:** an unpaced 16-way ten-skill burst had p95 **8.32 seconds** because company-level preparation serializes. Cold preparation took approximately **1.1–2.5 seconds** for one skill and **7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about **1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A concurrent whole-container sample was 3.45 GB. This does not establish suitability for a small-memory host or eliminate per-entrypoint native asset duplication. - **Upgrade without a manual data migration passed:** legacy snapshot JSON shrank from 22,465,691 bytes to `[]`, with a stored package and zero pending conversions. - **Recovery passed:** sixteen concurrent cold requests coalesced into two S3 GETs; full bytes/binary/executable modes matched; same-size corruption repaired; an injected storage failure preserved warm access and failed cold access safely. A disabled public-Git source denied recovery, and after the actual five-minute backoff its exact SHA reconstructed all 229 files. Cross-company descriptor access was denied. Private-Git credential revocation was not tested. - **Retention passed:** reclaimed 27,754,496 allocated cache bytes, deleted an unreferenced package row and both objects, preserved aged authored history, and expired/reacquired reconstructible Git content. A version pinned by the failed native run was correctly retained. Only owned fixture timestamps were aged. - **Browser UI passed:** authored skill creation, saved edit surviving reload, version history, and restoration of earlier content. - **Initial native attempt (superseded by the qualifying turns above):** one real run failed at Daytona allocation after five minutes (`setup_failed`), before any model execution. Provider cleanup also reported failure. That attempt does not count as a passing native turn. Its receipt and all evidence remain preserved; its cleanup later recorded success after normal restoration. - Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache permissions caused one setup HTTP 500 and the first UID-1000 soak to fail; exact fixture paths were repaired. The final soak, retention and recovery ran as application UID 1000. A fresh UID-1000 one-skill batch also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff and UI-wait failures remain recorded alongside their successful retries. - Private evidence includes raw timings, assertions, HTTP probes, screenshots and failure/cleanup receipts. Initial failed-campaign cleanup suspended and retained the owned TEST tenant and released the shared QA account lease at **15:22:41 UTC**. The tenant was subsequently resumed under its existing ownership for the native execution qualification above. Final cleanup completed at **15:53:09 UTC**: the account lease is released and the tenant is suspended/retained under the original October 30, 12:24:52 UTC retention deadline. Its separate stack lease remains held; no active or pending-cleanup environment lease remained. The original retention window is deliberately preserved even after successful continuation. - Earlier hosted-tested head: `a2e597d800eb989d565780e7aa947d5368676444`, integrated with master `929e840595`. The earlier merge conflicts were resolved at this head. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629) passed; all checks are complete with 55 successes, two neutral reports, two skips, and no failures. [Apex review 15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258) passed 5/5 with zero findings on this exact head. All review conversations are resolved. - Ten real-filesystem boundary cases and server typecheck pass locally for the security fix. Coverage includes cache roots, descendants, individual files, symlink aliases, recursive ancestors, a future concurrent cache fill, and allowed sibling directories. [Current-head private-project integration tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405) passed all 29 cases, including authorized and unauthorized imports of a cached private package. Full typecheck, build, server suites, all eight browser shards, and Runner checks passed in current-head CI. Local database tests could not run because PostgreSQL could not initialize; this limitation is recorded below. - The earlier generated migration preserved the newest five snapshots. The final integration renumbers its unchanged SQL to 0415. Migration safety/numbering checks pass. Its SQL is byte-for-byte identical to the previously verified idempotent skill migration; repeated application on this latest schema was blocked by the same local database startup failure. - The broader verification before the final migration-only rebase passed 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests, 52 focused UI tests, full workspace typecheck/build, token gates, and map checks. The linked complete CI run passed before the latest focused fixes. - A disposable MinIO instance passed multipart upload, verified read-back, and cold restoration of an 18 MB archive. - Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository retained five metadata queries and zero object reads. Median preparation was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated network delay each way. These are local measurements, not production latency guarantees. - No production instance was changed. Earlier verification failures and their disposition are retained below.
Earlier failures, review findings, fixes, and rerun evidence - The first CI pass found a stale inventory-span expectation and a missing repository-read span. Both are fixed; all three observability tests pass. One browser test copied page text into its YAML parser after reload; its first unchanged-head retry passed, but the race recurred after rebasing. The test now waits for the renamed URL and completed reload before copying the editor, retaining its original assertions. The earlier full local suite did not complete cleanly: it reported setup timeouts and assertions, while isolated relevant reruns passed. The complete current-head CI run closes that broad verification gap. No production instance was changed. - The first Apex review scored 2/5 and identified six issues. The fixes protect authored history even when it shares an in-flight Git upload, recheck upload expiry during orphan deletion, bound failed-upload cleanup, retain per-version kind/encoding metadata, add sanitized failure diagnostics, and correct the migration guide. The final focused run passes all 140 tests; the preceding eight-file run passed all 149 tests, including local storage and S3 diagnostics. Server typecheck and migration safety/numbering checks pass after the fixes. The second Apex review scored 1/5 and found unbounded maintenance storage calls and mismatched package/Git batch limits. Those are fixed with a 30-second cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and storage limits, and bounded Git read batches. The proposed manual lockfile edit was rejected with evidence of the enforced CI-owned lockfile policy. All 19 storage/recovery unit tests and server typechecking pass. The new database-backed cleanup/conversion tests were locally skipped because the host exhausted its PostgreSQL semaphore capacity (confirmed directly by initdb). That gap is closed by current-head CI: [package-store tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889) pass all 18 cases and [package-worker tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157) pass all 7, with no skips in either suite. The corrected YAML browser shard passes. A separate planning-fixture test failed its cleanup DELETE with an activity-log foreign key violation after its behavior assertions passed; its one unchanged-head retry passed. The third Apex review scored 1/5 and identified a one-slot pool deadlock, projected path limits, cleanup holding the preparation lock during archive reads, and a publication/expiry race. These are fixed with transaction-local reads, prefix-aware bounds, isolated verification, and shared package row locks with a publication grace. The separate Paperclip review also found stale historical inspection, eager history downloads, and conversion retries blocking sleep. Those are fixed. Local PostgreSQL capacity recovered: the new database regressions ran, including the one-slot fork, paused storage during a warm turn, and a locked publication during expiry. The expanded run passed 144/145 tests; its remaining assertion still expected file bodies from the version list and was updated to fetch the selected version. The corrected company-skill suite passes all 81 tests, and the final package/projection run passes all 27. All 52 focused UI tests pass after updating their query-provider fixtures; the first UI run exposed a fixture remount, and the follow-up exposed an editing syntax error, both corrected. Full workspace typecheck and build, token gates, and map checks pass. The sandbox initially blocked the build tool’s local IPC socket; the authorized build rerun passed. Master then claimed migration 0401. The branch is rebased on f8226b0515, the unchanged SQL is regenerated as 0402, and repeated application passed twice on the combined schema. Post-rebase workspace typecheck passes. All 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests pass on the combined head. All CI checks passed for head 9d80a3b52d. The fourth Apex review scored 3/5 and found a save/edit lock-order cycle, a retry retaining an uncommitted source ID in its recovery recipe, and a summary read racing conversion. Commit b7d9237c6a fixes all three. The new concurrency and retry tests pass within all 61 package/source tests, and server typecheck passes. Master then claimed migration 0402. The branch is now rebased on 65fa1cd617 and the unchanged SQL is regenerated as 0403. All 68 package, source, and worker regressions, server typecheck, migration safety/numbering, and repeated migration application pass on current head 62553671a0. Subsequent verification and review outcomes are recorded below. - The fifth Apex review scored 4/5 and found cached version reads unnecessarily depending on object storage and conversion reads missing the coordinator cancellation signal. Commit 1eb90d80be fixes both: cached reads verify bytes without the preparation lock; manifest and archive reads carry cancellation through conversion. The first focused run passed 70/71 tests, including all three new cases, but the existing diagnostic test observed no warning. Its fixture now sets an explicitly expired migration deadline; the final worker suite passes all nine cases. Server typecheck passes. - The sixth Apex review scored 4/5 with one finding: concurrent cached inventories were unbounded. Commit 4b7589e14c gives cached inventories and previews a two-reader/32-waiter limit independent of cold storage. The first expanded pressure test observed a third reader before the preview path had the same bound; both paths now share the local limit. All 73 focused cases pass, followed by all 29 package cases after the final overload adjustment; server typecheck passes. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestion to persist durability verification was considered: cleanup intentionally verifies current external bytes before discarding the last local authored copy, and the missing-object regression requires that check. Shared-package eviction happens on cold fills with at most two attempts and a ten-second budget. - The seventh Apex review reached 5/5 with one remaining P2: timeout-bearing manifest reads bypassed request coalescing. The security review also found repeated archive acquisition after a cold administrative read. Commit 79a80a39d2 shares manifest loads with per-caller cancellation and cancels the underlying work when the last caller leaves; coalesces identical cold version/file requests; and publishes verified restored packages into the shared cache. Publication uses a short nonblocking lock, while eviction verification runs outside it. The first expanded run passed 73/74 tests and exposed a missing originating DB for cache publication inside read-only inspection; transaction readers now pass that owner explicitly, keeping metadata on the transaction. The corrected run passed 75/75. A final eviction test initially expected the shared package to remain after only its separate runtime projection was reused; it now explicitly accesses the shared package during verification. All 32 final package tests pass, including independent caller cancellation, one acquisition across concurrent/repeated reads, and warm preparation during stalled eviction verification. Server typecheck passes. - The eighth Apex review scored 4/5 with one finding: decoded cold inventories outlived the storage semaphore while waiting for cleanup. Commit 9d7921d1ec adds a separate two-response/32-waiter lifetime bound held through cleanup, preserving the independent warm-read queue and forwarding cancellation through eviction verification. The new regression pauses cleanup after two responses decode, verifies a third archive stays queued, and completes warm preparation before releasing cleanup. All 77 package/source/worker cases and server typecheck pass. On the preceding head, serialized server shard 6/9 was interrupted with “The operation was canceled” after 57 seconds of tests; the log contains no failing assertion. Fresh latest-head CI reruns that shard along with the other gates. - The ninth Apex review scored 4/5 with one P1: cold responses could own both permits while waiting on a one-slot ordinary database pool held by a queued transaction reader. Commit a94f0e409f reads caller-scoped metadata before admission and isolates all acquisition/cleanup SQL on a private connection. The new real-PostgreSQL regression reproduces the deadlock against the previous head and passes with the fix; all 78 focused cases and server typecheck pass. The previous CI run completed with 54 successful checks, including both formerly interrupted/stuck serialized shards. - The tenth Apex review scored 4/5 with one P1: preloading legacy bodies before admission could retain many full repositories in queued requests. Commit 276e353a37 removes that preload and also bounds legacy manifest derivation/fallback, using private connections inside the I/O slot. All 80 focused tests pass; the final synchronized package suite passes all 36 tests and server typecheck passes. - Apex review 11 returned 5/5 with zero comments on 276e353a37. While it ran, master 8bc051245f added migration 0403, creating a metadata conflict. Merge commit e1783764d9 integrates master and generates 0404_clumsy_leo.sql; its skill SQL is byte-for-byte identical to the previously verified migration. The combined schema passes migration safety/numbering, repeated SQL application twice, all 80 focused PostgreSQL tests, full workspace typecheck, and full build. The preceding a94f0e409f CI run was superseded and cancelled while its final runner job was active; no old-head success is claimed for that run. - Final acceptance: Apex review 12 returned 5/5 with zero comments on e1783764d9. All 28 review threads are resolved. The final head has 55 successful checks, two neutral reports, two skipped checks, and no pending or failing checks; the PR workflow completed successfully. Superseded run 40789121902 was cancelled to free its workflow slot for the final head; normal cancellation did not finish promptly, so force cancellation completed it. - October 29 conflict resolution (867c00673b): master 929e840595 added installed-selection reuse and bundled relocation, routine cleanup notifications, a hardened YAML copy helper, and migrations through 0413. The merge retains those behaviors together with package read scoping and the public-selection fast path; restricted selections still reauthorize and the full row read rechecks scope. Generated migration 0414 retains the existing SQL unchanged. Workspace typecheck/build and migration checks pass. Local database verification is unavailable: initdb reports semget / “could not create semaphores: No space left on device”; all 168 cases skipped, and repeated migration application failed at startup. Docker returned HTTP 500 during container creation, so no Docker test pass is claimed. Current-head CI must close this gap. - Apex review 14 scored 3/5 with one P1: a local import could copy cached private package files into a company-wide skill. Commit a2e597d800 rejects cache trees and ancestors through a canonical filesystem boundary, including aliases and ancestors checked before a concurrent fill. Ten local filesystem tests and server typecheck pass. The existing private-project integration case now rejects package-cache imports for both authorized and unauthorized users. The preceding merge commit’s complete CI run passed, and logs confirm all 168 locally skipped cases executed successfully: package store 36, sources 35, package worker 9, runtime selection 4, observability 3, authoring 81. - Apex review 15 returned 5/5 with zero new findings on a2e597d800. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestions were considered: unexpected filesystem cleanup failures remain visible, the durable marker cannot replace checking that bytes still exist before deleting a last local authored copy, and administrative history batching remains outside the pre-turn optimization. The cache-path literal centralization suggestion is also optional; the current producer and guard use the same directory, covered by the new integration regression. All review conversations are resolved. - Earlier October 29 CI/Apex acceptance: a2e597d800 was mergeable at that check, Apex review 15 passed 5/5 with zero findings, and all review conversations were resolved. The hosted verification above records the later acceptance limits and renewed conflicts. Workflow 40884087629 completed successfully, with 55 successful checks, two neutral reports, two skips, and no pending or failed checks. The project-skill-sources job log confirms all 29 cases ran and passed, closing the latest integration gap. The preceding merge workflow also confirmed all 168 focused database cases ran and passed. No local PostgreSQL or Docker pass is claimed.
## Risks - First use after eviction can require storage or Git access. Revoked grants, removed sources, and unreachable commits can prevent reconstruction. Recovery never substitutes a newer commit. - Historical packages without a proven recipe keep durable bytes. Cleanup favors retention when a run pin cannot be interpreted safely. - New uploads and conversion use bounded work and persisted retries. Large initial conversions still add temporary storage and database load. - Optional cleanup can wait until the next server wake. Supported notifications assume one owning server process. - Per-entrypoint native tree duplication and cold preparation cost remain. The final ten-skill cold sample took 47.6 seconds on the shared host. - Generic authored edits can still hold an existing edit transaction while a package uploads. - Explicit file/folder deletion uses temporary disk space for rollback. An uncertain commit with a changed skill or unavailable database retains that backup and logs its location for recovery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and tests. The exact served model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DATABASE.md | 24 + doc/DEVELOPING.md | 5 + doc/PRODUCT.md | 6 + doc/SPEC-implementation.md | 14 +- doc/idle-sleep-safety.md | 11 + ...-28-skill-package-performance-results.json | 9741 +++++++++++++++++ doc/plans/2026-10-28-skill-package-storage.md | 444 + doc/recoverable-disk-cleanup.md | 10 +- doc/skill-package-storage.md | 307 + maps/agent-access-map/skills.md | 22 + maps/feature-map/skills.md | 14 + .../db/src/migrations/0415_zippy_zemo.sql | 81 + ...{0410_snapshot.json => 0415_snapshot.json} | 378 +- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/company_skills.ts | 74 +- packages/db/src/schema/index.ts | 3 +- packages/shared/src/index.ts | 1 + packages/shared/src/types/company-skill.ts | 9 +- packages/shared/src/types/index.ts | 1 + server/package.json | 2 + .../benchmark-native-skill-preparation.ts | 111 +- server/scripts/migrate-skill-packages.ts | 21 + server/scripts/verify-skill-package-s3.ts | 59 + .../__tests__/company-skills-service.test.ts | 14 +- .../src/__tests__/idle-sleep-safety.test.ts | 19 +- .../__tests__/project-skill-sources.test.ts | 17 +- .../src/__tests__/skill-package-store.test.ts | 986 ++ .../skill-repository-package.test.ts | 10 + server/src/__tests__/skill-sources.test.ts | 49 +- .../__tests__/storage-local-provider.test.ts | 15 +- .../__tests__/storage-s3-multipart.test.ts | 36 +- server/src/index.ts | 5 + server/src/routes/company-skills.ts | 4 +- server/src/services/company-deletion.ts | 1 + server/src/services/company-skills.ts | 374 +- .../services/customer-success-inspection.ts | 13 +- .../services/delivery-work-notifications.ts | 2 + server/src/services/file-manifest-cache.ts | 74 + server/src/services/github-skill-source.ts | 12 +- .../src/services/heartbeat/run-preparation.ts | 8 +- server/src/services/idle-sleep-safety.ts | 3 + .../runtime-asset-retention.test.ts | 57 + .../native-runtime/runtime-asset-retention.ts | 145 + .../native-runtime/runtime-context.test.ts | 18 + .../native-runtime/runtime-context.ts | 131 +- .../native-runtime/skill-asset-cache.ts | 46 + .../skill-local-import-boundary.test.ts | 42 + .../services/skill-local-import-boundary.ts | 22 + server/src/services/skill-package-cache.ts | 78 + server/src/services/skill-package-files.ts | 146 + .../src/services/skill-package-maintenance.ts | 136 + .../services/skill-package-recovery.test.ts | 63 + server/src/services/skill-package-recovery.ts | 75 + server/src/services/skill-package-runtime.ts | 120 + server/src/services/skill-package-store.ts | 674 ++ .../src/services/skill-package-work.test.ts | 176 + server/src/services/skill-package-work.ts | 191 + server/src/services/skill-snapshot.ts | 3 + server/src/services/skill-source-sync.ts | 6 +- server/src/services/skill-sources.ts | 68 +- server/src/services/skill-version-cache.ts | 77 +- .../src/services/skill-version-retention.ts | 3 +- server/src/storage/failure-cleanup.ts | 17 + server/src/storage/local-disk-provider.ts | 4 +- server/src/storage/s3-multipart.ts | 11 +- server/src/storage/s3-provider.ts | 2 + server/src/storage/service.ts | 26 +- server/src/storage/types.ts | 8 +- ui/src/api/companySkills.ts | 3 +- .../skill-studio/AgentsUsingSkillDialog.tsx | 6 +- ui/src/pages/CompanySkills.production.tsx | 32 +- ui/src/pages/CompanySkills.test.tsx | 24 +- ui/src/pages/CompanySkills.tsx | 32 +- ui/src/pages/SkillStudio.test.tsx | 1 + ui/src/pages/SkillStudio.tsx | 26 +- .../agent-skills/AgentSkillReleasePicker.tsx | 12 +- 76 files changed, 15125 insertions(+), 343 deletions(-) create mode 100644 doc/plans/2026-10-28-skill-package-performance-results.json create mode 100644 doc/plans/2026-10-28-skill-package-storage.md create mode 100644 doc/skill-package-storage.md create mode 100644 packages/db/src/migrations/0415_zippy_zemo.sql rename packages/db/src/migrations/meta/{0410_snapshot.json => 0415_snapshot.json} (99%) create mode 100644 server/scripts/migrate-skill-packages.ts create mode 100644 server/scripts/verify-skill-package-s3.ts create mode 100644 server/src/__tests__/skill-package-store.test.ts create mode 100644 server/src/services/file-manifest-cache.ts create mode 100644 server/src/services/native-runtime/runtime-asset-retention.test.ts create mode 100644 server/src/services/native-runtime/runtime-asset-retention.ts create mode 100644 server/src/services/native-runtime/skill-asset-cache.ts create mode 100644 server/src/services/skill-local-import-boundary.test.ts create mode 100644 server/src/services/skill-local-import-boundary.ts create mode 100644 server/src/services/skill-package-cache.ts create mode 100644 server/src/services/skill-package-files.ts create mode 100644 server/src/services/skill-package-maintenance.ts create mode 100644 server/src/services/skill-package-recovery.test.ts create mode 100644 server/src/services/skill-package-recovery.ts create mode 100644 server/src/services/skill-package-runtime.ts create mode 100644 server/src/services/skill-package-store.ts create mode 100644 server/src/services/skill-package-work.test.ts create mode 100644 server/src/services/skill-package-work.ts create mode 100644 server/src/storage/failure-cleanup.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index bf9f4a03333..687bbc5aaba 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -1252,6 +1252,30 @@ register before writing, or they can remain unseen until the next startup or another notification. Tests should verify both the producer and its outer transaction boundary. +### Installed skill packages + +`company_skill_repository_snapshots` retains its identity and company/content-hash +uniqueness. Despite its historical name, it can also hold editable skill packages. +Migration 0414 (`0414_sleepy_payback.sql`) adds `storage` (archive/manifest receipts) and `upload` (a bounded +pending attempt). `file_inventory` remains the legacy payload, defaulting to `[]`. +`company_skill_versions.package_snapshot_id` and `package_file_map` preserve the +public version file inventory without storing its contents again. Existing repository +snapshot/entrypoint references and version IDs remain unchanged. Reference indexes +support bounded orphan checks; restrictive FKs protect retained version packages. +Version mappings preserve kinds and encodings when identical bytes share a package. +Authored saves mark the storage receipt `durable`; retirement atomically respects that +marker even when the save joins a concurrent Git upload. + +`skill_package_deletions` is a durable object-deletion outbox populated by snapshot +update/delete triggers. Its company ID deliberately has no cascading company FK: +cleanup must retain exact object keys after company deletion. Upload promotion skips +retirement; replacing a pending token retires only the old attempt's unique keys. +The same migration adds exact-commit restore recipes and indexed conversion/maintenance +deadlines. A bounded coordinator worker automatically converts and clears verified +legacy payloads; no migration command is required. Warm reads exclude recipe manifests. +See [package storage](skill-package-storage.md) for acquisition, retention, compatibility +overrides, independent restore verification and coordinated backup policy. + ### Project repository skill synchronization `project_skill_sources` stores durable workspace import jobs and originating project diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 035d338de5c..a301752f7a6 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2885,6 +2885,11 @@ Environment overrides: non-numeric value falls back to the default. - `PAPERCLIP_WORKSPACE_COMPACTION_ENABLED=false` turns compaction off. Any other value keeps it on. +- `PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED=false` disables external writes and automatic + package maintenance for a coordinated rolling upgrade. Default: enabled. Legacy + conversion runs automatically in bounded batches, outside turn preparation. + Existing external packages remain readable with the override. See + [package storage and acquisition](skill-package-storage.md). - `PAPERCLIP_SKILL_VERSION_RETENTION_KEEP=` sets how many of the newest materialized versions of each skill stay on disk (see "Skill Version Snapshot Retention" above). The default is `0`. A negative, empty, or non-numeric value diff --git a/doc/PRODUCT.md b/doc/PRODUCT.md index 8bd41009632..2e2091de5bf 100644 --- a/doc/PRODUCT.md +++ b/doc/PRODUCT.md @@ -198,6 +198,12 @@ Missing references are advisory: a valid selected skill can still be imported. Unavailable files are listed, and existing installed dependencies are retained when a bounded refresh cannot preserve them. File limits and content validation still apply; importing never executes repository code. +Authored and unproven legacy versions retain an independent recovery copy. Unused Git +packages with an exact restore recipe may expire; later use requires access to that +original repository and commit. +External package storage keeps repository contents out of the control-plane database; +per-turn checks reuse verified local files while rechecking current skill access. +See [automatic storage and recovery](skill-package-storage.md). Project-derived skills inherit current project access. Access through any originating project is sufficient, including the responsible-user restrictions on agent execution. diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index a7f9edfe958..e397c78bcc4 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -2368,7 +2368,13 @@ unavailable. Preserve current ownership and newer-work fences. See on their snapshot. Importing does not install dependencies or configure host tools. - Stage and audit complete packages before publishing. Scripts are allowed through the existing content audit and never run on import. Persist binary bytes and executable - flags in immutable version inventories (legacy entries default to UTF-8/non-executable). + flags in immutable package snapshots (legacy entries default to UTF-8/non-executable). + Compatible readers support inline and external inventories. PostgreSQL retains + receipts, file mappings and exact Git restore recipes; the storage backend retains + bytes. No package is persisted without a valid selected skill. Legacy conversion + is automatic and bounded. Unused Git caches can expire and reacquire the recorded + commit after current authorization and hash checks; authored/unproven legacy bytes + remain durable. See [skill package storage](skill-package-storage.md). - Refresh is serialized per source, checks saved selection revisions, and publishes valid skills plus per-entry outcomes with required activity records in one transaction. Failed packages retain prior versions; access/scan/download failures retain all content. @@ -2380,8 +2386,10 @@ unavailable. Preserve current ownership and newer-work fences. See - Runtime version directories are disposable caches. An hourly, bounded sweep may evict non-current copies unused for one hour, preserving active and unresolved execution pins. Selection, materialization, and pin persistence serialize with - eviction across processes. Missing copies rebuild from immutable database bytes - without upstream access. See [recoverable disk cleanup](recoverable-disk-cleanup.md) + eviction across processes. Missing copies rebuild from immutable stored packages + or legacy database bytes without upstream access. Warm preparation uses manifests + and file identities, avoids repository-content SQL/object reads, and does not + reconcile unrelated library entries for known selected repository skills. See [recoverable disk cleanup](recoverable-disk-cleanup.md) for rollout controls, legacy grace periods, and interrupted cleanup recovery. - Manually selected sources retain reviewed selection, with new candidates checked by default and explicit folder exclusions retained. Upstream deletion keeps installed content with diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 8153e21253f..e535edcd569 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -445,3 +445,14 @@ timer remains; an empty buffer performs no database work. Existing Muse durable sleep blockers, external worker stop evidence, and ingress requirements remain unchanged. This change establishes database quiet for inactive Muse, not host sleep eligibility for a configured Muse connection. + +## Skill package maintenance + +The package delivery worker automatically converts legacy snapshots and versions. +Legacy contents remain authoritative until conversion is verified. Unfinished or +failed conversions, including rows retained while package storage is disabled, +may wait for the next ingress wake and do not block idle sleep. Their persisted +retry dates and package/object/native-asset retention timestamps are earliest +maintenance eligibility, not promised delivery deadlines. Workers reconstruct persisted rows and filesystem +access leases on startup, and use the existing coordinator's drain/standby/shutdown +controls. No external Cloud alarm or provider suspension is established by these tests. diff --git a/doc/plans/2026-10-28-skill-package-performance-results.json b/doc/plans/2026-10-28-skill-package-performance-results.json new file mode 100644 index 00000000000..fde86e8206f --- /dev/null +++ b/doc/plans/2026-10-28-skill-package-performance-results.json @@ -0,0 +1,9741 @@ +{ + "schema": "paperclip.skill-package-performance-results.v1", + "date": "2026-10-28", + "fixture": { + "baseFiles": 1349, + "baseJsonBytes": 22799129, + "kind": "synthetic repository; additional entrypoints added for 2- and 10-skill cases" + }, + "notes": [ + "Local synthetic PostgreSQL and filesystem benchmark, with native asset preparation, fresh agent authorization, 155 unrelated skills and 5 ms database delay in each direction. Not deployed Railway latency.", + "Twenty numbered warm samples; warm-new-service is separate. Peak RSS is the entire benchmark process, including application, fixture, and cold allocations, not incremental package memory.", + "Instance disk covers durable local object files plus extracted packages, runtime versions and native assets; filesystem block sharing/compression can affect allocated bytes.", + "Source hashes record the measured code. Subsequent changes bound cold manifest/eviction I/O and serialize version-history hydration; the measured warm runtime algorithm is unchanged." + ], + "measurements": [ + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "3f45986c6dd14f62b04e48b88d699b7c401d1ac877a894f7666696c84df8cc24", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 1, + "instanceDisk": { + "allocatedBytes": 96739328, + "fileBytes": 81013397, + "files": 4055 + }, + "repositoryHash": "5cedce49bfb80e1430df76f4b3889a1b20095545e9aeb575d1b037ab3c6ecdbe", + "repositoryFiles": 1349, + "runtimeFiles": 1350, + "runtimeBytes": 22377362, + "samples": [ + { + "label": "cold", + "elapsedMs": 3714.5453749999997, + "nativeAssetMs": 945.963667, + "sqlResponseBytes": 19994, + "objectGets": 2, + "objectReadBytes": 13450429, + "rssBytes": 839942144, + "cpuUserMs": 2924.79, + "cpuSystemMs": 2069.853, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 6 + }, + { + "label": "warm-1", + "elapsedMs": 270.38466699999844, + "nativeAssetMs": 27.069083000002138, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 866238464, + "cpuUserMs": 60.196, + "cpuSystemMs": 43.843, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 5 + }, + { + "label": "warm-2", + "elapsedMs": 267.416874999999, + "nativeAssetMs": 33.68354199999885, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 870596608, + "cpuUserMs": 56.781, + "cpuSystemMs": 40.212, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 5 + }, + { + "label": "warm-3", + "elapsedMs": 403.48920799999905, + "nativeAssetMs": 122.44841699999961, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 871415808, + "cpuUserMs": 56.459, + "cpuSystemMs": 51.775, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 5 + }, + { + "label": "warm-4", + "elapsedMs": 325.8029170000009, + "nativeAssetMs": 28.39483300000211, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 874086400, + "cpuUserMs": 53.772, + "cpuSystemMs": 52.706, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 5 + }, + { + "label": "warm-5", + "elapsedMs": 263.39350000000195, + "nativeAssetMs": 27.145415999999386, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 877297664, + "cpuUserMs": 46.555, + "cpuSystemMs": 58.031, + "processPeakRssBytes": 877297664, + "rootPoolQueries": 5 + }, + { + "label": "warm-6", + "elapsedMs": 258.43724999999904, + "nativeAssetMs": 24.80045799999789, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 877412352, + "cpuUserMs": 45.271, + "cpuSystemMs": 49.475, + "processPeakRssBytes": 877412352, + "rootPoolQueries": 5 + }, + { + "label": "warm-7", + "elapsedMs": 249.36533299999792, + "nativeAssetMs": 21.476291999999376, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 882917376, + "cpuUserMs": 40.674, + "cpuSystemMs": 57.454, + "processPeakRssBytes": 882917376, + "rootPoolQueries": 5 + }, + { + "label": "warm-8", + "elapsedMs": 238.76808299999902, + "nativeAssetMs": 19.84570800000074, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 885981184, + "cpuUserMs": 36.699, + "cpuSystemMs": 38.557, + "processPeakRssBytes": 885981184, + "rootPoolQueries": 5 + }, + { + "label": "warm-9", + "elapsedMs": 254.60462499999994, + "nativeAssetMs": 27.14166699999987, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886538240, + "cpuUserMs": 43.802, + "cpuSystemMs": 42.745, + "processPeakRssBytes": 886538240, + "rootPoolQueries": 5 + }, + { + "label": "warm-10", + "elapsedMs": 238.2076249999991, + "nativeAssetMs": 18.28241700000217, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886571008, + "cpuUserMs": 35.114, + "cpuSystemMs": 38.915, + "processPeakRssBytes": 886571008, + "rootPoolQueries": 5 + }, + { + "label": "warm-11", + "elapsedMs": 237.13320799999929, + "nativeAssetMs": 17.53529200000048, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886587392, + "cpuUserMs": 32.11, + "cpuSystemMs": 40.532, + "processPeakRssBytes": 886587392, + "rootPoolQueries": 5 + }, + { + "label": "warm-12", + "elapsedMs": 254.8944589999992, + "nativeAssetMs": 21.32683299999917, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886734848, + "cpuUserMs": 35.116, + "cpuSystemMs": 54.695, + "processPeakRssBytes": 886734848, + "rootPoolQueries": 5 + }, + { + "label": "warm-13", + "elapsedMs": 246.8540839999987, + "nativeAssetMs": 21.06829099999959, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886816768, + "cpuUserMs": 50.899, + "cpuSystemMs": 42.277, + "processPeakRssBytes": 886816768, + "rootPoolQueries": 5 + }, + { + "label": "warm-14", + "elapsedMs": 243.9827499999992, + "nativeAssetMs": 19.04475000000093, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886833152, + "cpuUserMs": 35.215, + "cpuSystemMs": 43.516, + "processPeakRssBytes": 886833152, + "rootPoolQueries": 5 + }, + { + "label": "warm-15", + "elapsedMs": 238.93862499999886, + "nativeAssetMs": 18.254375000000437, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 861929472, + "cpuUserMs": 32.799, + "cpuSystemMs": 37.347, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-16", + "elapsedMs": 242.26750000000175, + "nativeAssetMs": 17.968958000001294, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 862502912, + "cpuUserMs": 37.265, + "cpuSystemMs": 36.262, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-17", + "elapsedMs": 231.48641700000007, + "nativeAssetMs": 15.712750000002416, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 864927744, + "cpuUserMs": 31.139, + "cpuSystemMs": 38.036, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-18", + "elapsedMs": 233.1889169999995, + "nativeAssetMs": 15.838167000001704, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 867467264, + "cpuUserMs": 27.639, + "cpuSystemMs": 38.248, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-19", + "elapsedMs": 238.24141699999745, + "nativeAssetMs": 14.957500000000437, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 867876864, + "cpuUserMs": 29.939, + "cpuSystemMs": 33.88, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-20", + "elapsedMs": 230.42920900000172, + "nativeAssetMs": 16.502959000001283, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 868384768, + "cpuUserMs": 28.371, + "cpuSystemMs": 38.99, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-new-service", + "elapsedMs": 232.0073749999974, + "nativeAssetMs": 15.748999999999796, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 869056512, + "cpuUserMs": 28.423, + "cpuSystemMs": 34.852, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + } + ], + "warmSummary": { + "count": 20, + "p50Ms": 245.41841699999895, + "p95Ms": 325.8029170000009, + "minMs": 230.42920900000172, + "maxMs": 403.48920799999905, + "medianCpuMs": 82.639 + } + }, + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "3f45986c6dd14f62b04e48b88d699b7c401d1ac877a894f7666696c84df8cc24", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 2, + "instanceDisk": { + "allocatedBytes": 152195072, + "fileBytes": 125969570, + "files": 6762 + }, + "repositoryHash": "fe0771eb3fea47e84689b61659ad5e4c320fb5ba77ca289bf8948220eb8acb83", + "repositoryFiles": 1350, + "runtimeFiles": 1351, + "runtimeBytes": 22377459, + "samples": [ + { + "label": "cold", + "elapsedMs": 6328.536042000001, + "nativeAssetMs": 1185.029708, + "sqlResponseBytes": 21018, + "objectGets": 2, + "objectReadBytes": 13450654, + "rssBytes": 864239616, + "cpuUserMs": 3393.371, + "cpuSystemMs": 3749.683, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 6 + }, + { + "label": "warm-1", + "elapsedMs": 296.3216250000005, + "nativeAssetMs": 45.2325409999994, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 897777664, + "cpuUserMs": 88.313, + "cpuSystemMs": 91.375, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-2", + "elapsedMs": 280.3572920000006, + "nativeAssetMs": 46.95887500000026, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 901021696, + "cpuUserMs": 78.114, + "cpuSystemMs": 95.396, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-3", + "elapsedMs": 276.8585409999996, + "nativeAssetMs": 42.61787499999991, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 901251072, + "cpuUserMs": 70.772, + "cpuSystemMs": 91.847, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-4", + "elapsedMs": 284.81624999999985, + "nativeAssetMs": 46.87012499999946, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 901365760, + "cpuUserMs": 67.437, + "cpuSystemMs": 97.816, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-5", + "elapsedMs": 283.0642910000006, + "nativeAssetMs": 44.04358300000058, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 901709824, + "cpuUserMs": 68.311, + "cpuSystemMs": 93.898, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-6", + "elapsedMs": 283.67491599999994, + "nativeAssetMs": 46.34595900000022, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 902168576, + "cpuUserMs": 69.452, + "cpuSystemMs": 91.08, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-7", + "elapsedMs": 302.01966699999684, + "nativeAssetMs": 47.38970799999879, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 902905856, + "cpuUserMs": 81.11, + "cpuSystemMs": 99.893, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-8", + "elapsedMs": 333.87829200000124, + "nativeAssetMs": 57.75833400000192, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 903921664, + "cpuUserMs": 82.534, + "cpuSystemMs": 110.13, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-9", + "elapsedMs": 332.312582999999, + "nativeAssetMs": 59.70783400000073, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 905183232, + "cpuUserMs": 86.169, + "cpuSystemMs": 133.898, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-10", + "elapsedMs": 309.3872909999991, + "nativeAssetMs": 57.14895799999795, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 905838592, + "cpuUserMs": 78.898, + "cpuSystemMs": 125.806, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-11", + "elapsedMs": 313.51016700000037, + "nativeAssetMs": 62.051665999999386, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 906330112, + "cpuUserMs": 81.977, + "cpuSystemMs": 130.557, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-12", + "elapsedMs": 326.54258400000253, + "nativeAssetMs": 74.06966600000305, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 906952704, + "cpuUserMs": 72.584, + "cpuSystemMs": 102.054, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-13", + "elapsedMs": 294.62420900000143, + "nativeAssetMs": 50.54508299999725, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 907395072, + "cpuUserMs": 71.091, + "cpuSystemMs": 104.148, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-14", + "elapsedMs": 291.966292000001, + "nativeAssetMs": 49.89749999999913, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 907902976, + "cpuUserMs": 75.121, + "cpuSystemMs": 98.96, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-15", + "elapsedMs": 278.43558400000256, + "nativeAssetMs": 43.16858299999876, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 908443648, + "cpuUserMs": 70.947, + "cpuSystemMs": 91.63, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-16", + "elapsedMs": 280.7341250000027, + "nativeAssetMs": 43.70062499999767, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 908951552, + "cpuUserMs": 65.543, + "cpuSystemMs": 91.628, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-17", + "elapsedMs": 307.03354200000103, + "nativeAssetMs": 58.58600000000297, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 909524992, + "cpuUserMs": 75.961, + "cpuSystemMs": 114.523, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-18", + "elapsedMs": 304.9934580000008, + "nativeAssetMs": 53.50358300000153, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 909901824, + "cpuUserMs": 70.672, + "cpuSystemMs": 105.85, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-19", + "elapsedMs": 298.8248749999984, + "nativeAssetMs": 52.481667000000016, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 910409728, + "cpuUserMs": 69.146, + "cpuSystemMs": 112.195, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-20", + "elapsedMs": 325.0633339999986, + "nativeAssetMs": 52.81354099999953, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 910901248, + "cpuUserMs": 65.785, + "cpuSystemMs": 106.417, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-new-service", + "elapsedMs": 301.2625000000007, + "nativeAssetMs": 53.47154199999932, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 911409152, + "cpuUserMs": 71.72, + "cpuSystemMs": 112.835, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + } + ], + "warmSummary": { + "count": 20, + "p50Ms": 297.57324999999946, + "p95Ms": 332.312582999999, + "minMs": 276.8585409999996, + "maxMs": 333.87829200000124, + "medianCpuMs": 174.93849999999998 + } + }, + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "3f45986c6dd14f62b04e48b88d699b7c401d1ac877a894f7666696c84df8cc24", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 10, + "instanceDisk": { + "allocatedBytes": 596471808, + "fileBytes": 485644240, + "files": 28562 + }, + "repositoryHash": "8905e1b5adbdbc1f4be27131dcc3065dddbe615f73e90f8e0ecd9a6349feab43", + "repositoryFiles": 1358, + "runtimeFiles": 1359, + "runtimeBytes": 22378235, + "samples": [ + { + "label": "cold", + "elapsedMs": 11038.252875000002, + "nativeAssetMs": 2835.388499999999, + "sqlResponseBytes": 29242, + "objectGets": 2, + "objectReadBytes": 13452324, + "rssBytes": 1175388160, + "cpuUserMs": 10688.652, + "cpuSystemMs": 12125.498, + "processPeakRssBytes": 1175388160, + "rootPoolQueries": 6 + }, + { + "label": "warm-1", + "elapsedMs": 509.059874999999, + "nativeAssetMs": 206.29983300000094, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1200144384, + "cpuUserMs": 274.241, + "cpuSystemMs": 450.411, + "processPeakRssBytes": 1200144384, + "rootPoolQueries": 5 + }, + { + "label": "warm-2", + "elapsedMs": 563.2716670000009, + "nativeAssetMs": 229.46791700000176, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1205272576, + "cpuUserMs": 306.689, + "cpuSystemMs": 498.681, + "processPeakRssBytes": 1205272576, + "rootPoolQueries": 5 + }, + { + "label": "warm-3", + "elapsedMs": 551.9728749999995, + "nativeAssetMs": 224.04054199999882, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1208942592, + "cpuUserMs": 287.219, + "cpuSystemMs": 511.782, + "processPeakRssBytes": 1208942592, + "rootPoolQueries": 5 + }, + { + "label": "warm-4", + "elapsedMs": 540.7157499999994, + "nativeAssetMs": 213.42754199999763, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1212006400, + "cpuUserMs": 276.609, + "cpuSystemMs": 480.152, + "processPeakRssBytes": 1212006400, + "rootPoolQueries": 5 + }, + { + "label": "warm-5", + "elapsedMs": 578.5833329999987, + "nativeAssetMs": 238.3347500000018, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216577536, + "cpuUserMs": 304.291, + "cpuSystemMs": 483.345, + "processPeakRssBytes": 1216577536, + "rootPoolQueries": 5 + }, + { + "label": "warm-6", + "elapsedMs": 524.1193330000024, + "nativeAssetMs": 206.63066600000093, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1220493312, + "cpuUserMs": 262.339, + "cpuSystemMs": 476.115, + "processPeakRssBytes": 1220493312, + "rootPoolQueries": 5 + }, + { + "label": "warm-7", + "elapsedMs": 522.1358340000006, + "nativeAssetMs": 203.664249999998, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1223901184, + "cpuUserMs": 255.151, + "cpuSystemMs": 504.054, + "processPeakRssBytes": 1223901184, + "rootPoolQueries": 5 + }, + { + "label": "warm-8", + "elapsedMs": 536.1341250000005, + "nativeAssetMs": 223.24095900000248, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1228275712, + "cpuUserMs": 269.799, + "cpuSystemMs": 495.782, + "processPeakRssBytes": 1228275712, + "rootPoolQueries": 5 + }, + { + "label": "warm-9", + "elapsedMs": 527.7411670000001, + "nativeAssetMs": 207.70304100000067, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1232420864, + "cpuUserMs": 275.873, + "cpuSystemMs": 475.242, + "processPeakRssBytes": 1232420864, + "rootPoolQueries": 5 + }, + { + "label": "warm-10", + "elapsedMs": 535.4337919999998, + "nativeAssetMs": 221.5627080000013, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1227620352, + "cpuUserMs": 277.786, + "cpuSystemMs": 552.621, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-11", + "elapsedMs": 585.1106249999975, + "nativeAssetMs": 240.04220799999894, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1228800000, + "cpuUserMs": 311.691, + "cpuSystemMs": 525.373, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-12", + "elapsedMs": 582.450458000003, + "nativeAssetMs": 240.56300000000192, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1228947456, + "cpuUserMs": 319.294, + "cpuSystemMs": 517.277, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-13", + "elapsedMs": 1249.3626670000012, + "nativeAssetMs": 432.052208000001, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1229373440, + "cpuUserMs": 359.244, + "cpuSystemMs": 551.464, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-14", + "elapsedMs": 599.4650000000001, + "nativeAssetMs": 259.7930839999972, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1229897728, + "cpuUserMs": 321.251, + "cpuSystemMs": 539.465, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-15", + "elapsedMs": 674.1641249999993, + "nativeAssetMs": 298.9093750000029, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1229963264, + "cpuUserMs": 368.737, + "cpuSystemMs": 617.768, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-16", + "elapsedMs": 636.989166000003, + "nativeAssetMs": 294.70037500000035, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1230864384, + "cpuUserMs": 345.605, + "cpuSystemMs": 578.557, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-17", + "elapsedMs": 613.3584579999988, + "nativeAssetMs": 264.15899999999965, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1231388672, + "cpuUserMs": 336.042, + "cpuSystemMs": 511.512, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-18", + "elapsedMs": 592.2004589999997, + "nativeAssetMs": 239.67762499999662, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216462848, + "cpuUserMs": 427.338, + "cpuSystemMs": 523.325, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-19", + "elapsedMs": 584.3585830000011, + "nativeAssetMs": 250.61229200000525, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216462848, + "cpuUserMs": 316.455, + "cpuSystemMs": 520.297, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-20", + "elapsedMs": 566.4432920000036, + "nativeAssetMs": 226.26541700000234, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216462848, + "cpuUserMs": 297.778, + "cpuSystemMs": 511.475, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-new-service", + "elapsedMs": 552.4209159999955, + "nativeAssetMs": 222.31191699999908, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216643072, + "cpuUserMs": 292.34, + "cpuSystemMs": 521.954, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + } + ], + "warmSummary": { + "count": 20, + "p50Ms": 572.5133125000011, + "p95Ms": 674.1641249999993, + "minMs": 509.059874999999, + "maxMs": 1249.3626670000012, + "medianCpuMs": 819.8299999999999 + } + } + ], + "s3Proof": { + "server": "MinIO RELEASE.2025-04-22T22-12-26Z built from official source", + "files": 22, + "archiveBytes": 18008626, + "decodedBytes": 18000095, + "multipartRoundTrip": true, + "coldRestore": true, + "verifiedDurable": true + }, + "automaticLifecycleVerification": { + "date": "2026-10-28", + "notes": [ + "Final implementation with automatic conversion, exact-commit acquisition, native asset expiry and repair.", + "PAPERCLIP_TEST_POSTGRES_TEMPLATE=0 bypassed a shared test-template startup failure; fresh disposable PostgreSQL clusters were used.", + "Same synthetic input, 5 ms emulated network delay each way, authorization, unrelated-library shape, and native asset delivery as the earlier measurements.", + "Host was shared with other work. Timings are local regression samples, not production latency guarantees." + ], + "summaries": [ + { + "selectedSkills": 1, + "warmSamples": 20, + "medianMs": 256.1131045000002, + "p95Ms": 302.01637499999924, + "rootPoolQueries": [ + 5 + ], + "sqlResponseBytes": [ + 19423 + ], + "objectGets": 0, + "objectReadBytes": 0, + "coldMs": 5521.755166999999, + "sourceChangesHash": "46a4a05b61b399d18d1009085b6e54e9853563af98b3580318fb69290f0e05f1" + }, + { + "selectedSkills": 10, + "warmSamples": 20, + "medianMs": 690.6289374999978, + "p95Ms": 815.8250839999964, + "rootPoolQueries": [ + 5 + ], + "sqlResponseBytes": [ + 28969 + ], + "objectGets": 0, + "objectReadBytes": 0, + "coldMs": 47645.3685, + "sourceChangesHash": "46a4a05b61b399d18d1009085b6e54e9853563af98b3580318fb69290f0e05f1" + } + ], + "measurements": [ + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "46a4a05b61b399d18d1009085b6e54e9853563af98b3580318fb69290f0e05f1", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 1, + "instanceDisk": { + "allocatedBytes": 96739328, + "fileBytes": 81013397, + "files": 4056 + }, + "repositoryHash": "5cedce49bfb80e1430df76f4b3889a1b20095545e9aeb575d1b037ab3c6ecdbe", + "repositoryFiles": 1349, + "runtimeFiles": 1350, + "runtimeBytes": 22377362, + "samples": [ + { + "label": "cold", + "elapsedMs": 5521.755166999999, + "nativeAssetMs": 2862.993250000003, + "sqlResponseBytes": 20129, + "objectGets": 2, + "objectReadBytes": 13450429, + "rssBytes": 839041024, + "cpuUserMs": 2907.478, + "cpuSystemMs": 2180.612, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 6, + "traceId": "d930e609f6524d003325bbab3cd24fff", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 123.332, + "spanId": "575fff8ba56923bc", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + }, + { + "name": "skills.rows", + "durationMs": 24.519875, + "spanId": "691e337e466035ff", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + }, + { + "name": "skills.version.read", + "durationMs": 24.449625, + "spanId": "52d8b8322c1f9fbf", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.714625, + "spanId": "7b3663ddbf76629f", + "parentSpanId": "2271436e8b6ebccf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 2415.632125, + "spanId": "2271436e8b6ebccf", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + }, + { + "name": "skills.consume", + "durationMs": 2884.642125, + "spanId": "91f51464e883bbeb", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + } + ] + }, + { + "label": "warm-1", + "elapsedMs": 277.8793340000011, + "nativeAssetMs": 25.730290999999852, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 859815936, + "cpuUserMs": 68.572, + "cpuSystemMs": 48.415, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "17f51b0a84f58bf1add1d37a9529d0d9", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 105.477125, + "spanId": "e61340dcd3f750f1", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + }, + { + "name": "skills.rows", + "durationMs": 22.589, + "spanId": "bedf6f5708ca6865", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + }, + { + "name": "skills.version.read", + "durationMs": 24.176708, + "spanId": "b01f550b571c79df", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.274042, + "spanId": "1f6045577c1ce282", + "parentSpanId": "0f478bd06304aa65", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 38.816041, + "spanId": "0f478bd06304aa65", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + }, + { + "name": "skills.consume", + "durationMs": 50.267792, + "spanId": "3f191c4b1c5f2655", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + } + ] + }, + { + "label": "warm-2", + "elapsedMs": 258.88416700000016, + "nativeAssetMs": 22.96679100000256, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860487680, + "cpuUserMs": 60.559, + "cpuSystemMs": 47.455, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "baa58a8abe13ada2e7b5f888f1b543ce", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.164125, + "spanId": "039f5ed5d1f5b830", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + }, + { + "name": "skills.rows", + "durationMs": 23.490791, + "spanId": "e621641b17743498", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + }, + { + "name": "skills.version.read", + "durationMs": 21.576416, + "spanId": "4db591c9e0feecb3", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.304541, + "spanId": "6878c2b616698075", + "parentSpanId": "35aeb337e724e325", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 38.412708, + "spanId": "35aeb337e724e325", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + }, + { + "name": "skills.consume", + "durationMs": 46.015041, + "spanId": "bbb3f00f9b4bfdf3", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + } + ] + }, + { + "label": "warm-3", + "elapsedMs": 256.2706670000007, + "nativeAssetMs": 20.635957999998936, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860504064, + "cpuUserMs": 52.112, + "cpuSystemMs": 39.375, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "e5ff1facc7bda4e65d75f4e7549bff8c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 92.212625, + "spanId": "dca51402638b670c", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + }, + { + "name": "skills.rows", + "durationMs": 22.964166, + "spanId": "0439f29191eac439", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.306042, + "spanId": "466f10562fc4caf7", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.067, + "spanId": "9f9e84fd9ef963eb", + "parentSpanId": "d5fbca670b5f91dd", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 37.148208, + "spanId": "d5fbca670b5f91dd", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + }, + { + "name": "skills.consume", + "durationMs": 43.428084, + "spanId": "9f55673f9ed3273f", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + } + ] + }, + { + "label": "warm-4", + "elapsedMs": 252.9054580000011, + "nativeAssetMs": 21.87449999999808, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860553216, + "cpuUserMs": 48.459, + "cpuSystemMs": 39.237, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "ef8a05c329b3e2336fc3486592050728", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 89.006375, + "spanId": "e85a4aadbd4feccc", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + }, + { + "name": "skills.rows", + "durationMs": 22.330958, + "spanId": "f850ca66e27a29b1", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.321, + "spanId": "0d26d736bd91d71d", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.522458, + "spanId": "1d8bbd402e3d0c01", + "parentSpanId": "6ceccd78ea3bc47e", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 37.512292, + "spanId": "6ceccd78ea3bc47e", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + }, + { + "name": "skills.consume", + "durationMs": 44.858833, + "spanId": "37c073e9f09fed4d", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + } + ] + }, + { + "label": "warm-5", + "elapsedMs": 244.24670799999876, + "nativeAssetMs": 19.243334000002505, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860618752, + "cpuUserMs": 38.593, + "cpuSystemMs": 34.984, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "2bd5af6006769dd2721612d19cf0e7e5", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 86.527541, + "spanId": "27cb1035d28b6263", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + }, + { + "name": "skills.rows", + "durationMs": 22.759084, + "spanId": "e542b8f3f97b7625", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.41175, + "spanId": "f02d61a8f91295fd", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.862125, + "spanId": "9435bcb01b542105", + "parentSpanId": "f973c621f62205ec", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 35.202375, + "spanId": "f973c621f62205ec", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + }, + { + "name": "skills.consume", + "durationMs": 41.587958, + "spanId": "0af0696783146f45", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + } + ] + }, + { + "label": "warm-6", + "elapsedMs": 246.31941700000243, + "nativeAssetMs": 20.167625000001863, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860651520, + "cpuUserMs": 40.133, + "cpuSystemMs": 35.96, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "e9d381b984412682ee0d054785f5bc07", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 88.371584, + "spanId": "d7f9a9d6d1f6a9f7", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + }, + { + "name": "skills.rows", + "durationMs": 23.367584, + "spanId": "83b30eb0d37f3d43", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + }, + { + "name": "skills.version.read", + "durationMs": 21.615917, + "spanId": "20f08029124c778a", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.984708, + "spanId": "2ff9b56bf7d54e5d", + "parentSpanId": "6f5238c569424956", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 35.582, + "spanId": "6f5238c569424956", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + }, + { + "name": "skills.consume", + "durationMs": 43.176875, + "spanId": "0f1bcf11f3cb5b88", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + } + ] + }, + { + "label": "warm-7", + "elapsedMs": 260.0483329999988, + "nativeAssetMs": 20.238124999999854, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860651520, + "cpuUserMs": 40.479, + "cpuSystemMs": 42.819, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "0e7b563ae50b82c18c950d14822a0238", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 87.678792, + "spanId": "1a0f8d270b4ae1c2", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + }, + { + "name": "skills.rows", + "durationMs": 23.014667, + "spanId": "61e2f4ed73027123", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.224416, + "spanId": "a27fd4a5b6d6f50d", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + }, + { + "name": "skills.version.verify", + "durationMs": 23.332666, + "spanId": "20a9c335269146ad", + "parentSpanId": "6ae03d2c64c54861", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 50.015208, + "spanId": "6ae03d2c64c54861", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + }, + { + "name": "skills.consume", + "durationMs": 41.842834, + "spanId": "ac49533cdf6e0aba", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + } + ] + }, + { + "label": "warm-8", + "elapsedMs": 296.3832079999993, + "nativeAssetMs": 23.272417000000132, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860717056, + "cpuUserMs": 44.182, + "cpuSystemMs": 40.068, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "b313a786abc8784f76eebb04f2b65fd1", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.983125, + "spanId": "01327dfcea49dd05", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + }, + { + "name": "skills.rows", + "durationMs": 23.028041, + "spanId": "3419633149135de9", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.623292, + "spanId": "7a5993593c7b5a2a", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 39.360792, + "spanId": "86ff67922c241ade", + "parentSpanId": "0389067fc98ed3fd", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 76.266167, + "spanId": "0389067fc98ed3fd", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + }, + { + "name": "skills.consume", + "durationMs": 46.797167, + "spanId": "b6de6fa36685f548", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + } + ] + }, + { + "label": "warm-9", + "elapsedMs": 249.17170800000167, + "nativeAssetMs": 21.109082999999373, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 861306880, + "cpuUserMs": 44.78, + "cpuSystemMs": 38.28, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "8b174d18f9de1da799551b952d21375c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.344708, + "spanId": "050896cedad88644", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + }, + { + "name": "skills.rows", + "durationMs": 23.4595, + "spanId": "099e4789f58db9be", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.737334, + "spanId": "899d373a466f0ee4", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.478084, + "spanId": "c003f06a34d9fd6b", + "parentSpanId": "656625d058c3f76c", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 33.079208, + "spanId": "656625d058c3f76c", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + }, + { + "name": "skills.consume", + "durationMs": 44.442958, + "spanId": "16ada23b6310af02", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + } + ] + }, + { + "label": "warm-10", + "elapsedMs": 246.82358399999794, + "nativeAssetMs": 18.57062500000029, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 832356352, + "cpuUserMs": 37.233, + "cpuSystemMs": 34.641, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "f8bda941e19a7e3e55eb6ba599c8dc64", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.506625, + "spanId": "a89bd12424ddcc52", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + }, + { + "name": "skills.rows", + "durationMs": 24.072709, + "spanId": "1c7e14d1d64a7c1b", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.888542, + "spanId": "1aae1cba9874b9a8", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.823833, + "spanId": "e6dbd7401dd06462", + "parentSpanId": "6cf81c1ffc2675ec", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 33.593625, + "spanId": "6cf81c1ffc2675ec", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + }, + { + "name": "skills.consume", + "durationMs": 41.207167, + "spanId": "5e54082e0a3da96d", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + } + ] + }, + { + "label": "warm-11", + "elapsedMs": 247.22716699999728, + "nativeAssetMs": 18.73529200000121, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 832847872, + "cpuUserMs": 34.961, + "cpuSystemMs": 35.58, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "bc43ae232b084940d1d2916542b90f38", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.696958, + "spanId": "f761332966b03494", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + }, + { + "name": "skills.rows", + "durationMs": 23.352083, + "spanId": "96bf5714ed810b6c", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.087541, + "spanId": "e1f97ad53c5fa76f", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.558459, + "spanId": "6f379bc6aebb7259", + "parentSpanId": "5d12ce05f1977b62", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 33.8355, + "spanId": "5d12ce05f1977b62", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + }, + { + "name": "skills.consume", + "durationMs": 41.15275, + "spanId": "aba50da7f24b77f6", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + } + ] + }, + { + "label": "warm-12", + "elapsedMs": 246.921957999999, + "nativeAssetMs": 20.732458999998926, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 834371584, + "cpuUserMs": 37.99, + "cpuSystemMs": 39.364, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "104eae77376a1d6eed770793f6005b80", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.51825, + "spanId": "0d615cb98e361910", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + }, + { + "name": "skills.rows", + "durationMs": 23.632041, + "spanId": "cb25e039dd457704", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + }, + { + "name": "skills.version.read", + "durationMs": 21.565334, + "spanId": "94853b0c375376ff", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.611708, + "spanId": "b9c7020271c407dc", + "parentSpanId": "f4e9f25aad1cbc1c", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 32.759458, + "spanId": "f4e9f25aad1cbc1c", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + }, + { + "name": "skills.consume", + "durationMs": 44.410834, + "spanId": "8746f9a0a7a83f1b", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + } + ] + }, + { + "label": "warm-13", + "elapsedMs": 248.8155000000006, + "nativeAssetMs": 20.55420899999808, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 835338240, + "cpuUserMs": 49.511, + "cpuSystemMs": 40.047, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "a4e34ea74e542a06498eb2c0bfdb7d78", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 87.402583, + "spanId": "6bd7e492a84a3d62", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + }, + { + "name": "skills.rows", + "durationMs": 23.381709, + "spanId": "69786f2248b675e9", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + }, + { + "name": "skills.version.read", + "durationMs": 21.377708, + "spanId": "e761580f607cff81", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.884625, + "spanId": "cebe63bb7ed1775f", + "parentSpanId": "532261ccffb31019", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.704917, + "spanId": "532261ccffb31019", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + }, + { + "name": "skills.consume", + "durationMs": 43.189333, + "spanId": "6fcefc7fab230e8c", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + } + ] + }, + { + "label": "warm-14", + "elapsedMs": 254.31112500000017, + "nativeAssetMs": 21.776166999999987, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 837812224, + "cpuUserMs": 44.22, + "cpuSystemMs": 37.706, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "09058904f6838d35fab91829f2cede1c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 91.188416, + "spanId": "48bb96823338e726", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + }, + { + "name": "skills.rows", + "durationMs": 21.949208, + "spanId": "d42c2544cc22b1b1", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.344958, + "spanId": "1d2d49b58286e326", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.674125, + "spanId": "8ae309fa7a8ae97c", + "parentSpanId": "bc3c6b564be907eb", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.717125, + "spanId": "bc3c6b564be907eb", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + }, + { + "name": "skills.consume", + "durationMs": 45.310375, + "spanId": "9eb0e8a23aaa004c", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + } + ] + }, + { + "label": "warm-15", + "elapsedMs": 256.34162500000093, + "nativeAssetMs": 21.958875000000262, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 838369280, + "cpuUserMs": 40.477, + "cpuSystemMs": 40.68, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "85e9e33244624553ab1ef014fcbc202b", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 91.318042, + "spanId": "bb22f11d4bbe41e8", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + }, + { + "name": "skills.rows", + "durationMs": 24.197083, + "spanId": "c799b894086f66c6", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.747458, + "spanId": "9cb5bf472976a8e2", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.437792, + "spanId": "53faecf9fc592d09", + "parentSpanId": "241fd52adb2211bf", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.593166, + "spanId": "241fd52adb2211bf", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + }, + { + "name": "skills.consume", + "durationMs": 44.317041, + "spanId": "41fb8559f4d29765", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + } + ] + }, + { + "label": "warm-16", + "elapsedMs": 361.1522499999992, + "nativeAssetMs": 56.873374999999214, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 838828032, + "cpuUserMs": 45.728, + "cpuSystemMs": 34.21, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "5f24f8506e0b6fddedde523af9afa4f5", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 88.929625, + "spanId": "7a196c77e19eff83", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + }, + { + "name": "skills.rows", + "durationMs": 22.999291, + "spanId": "cd320c350c1f601a", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.893541, + "spanId": "34dccbf348b6dd3f", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + }, + { + "name": "skills.version.verify", + "durationMs": 56.2555, + "spanId": "0423cf859794a3cf", + "parentSpanId": "bc4ec6d07d341dfd", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 105.091542, + "spanId": "bc4ec6d07d341dfd", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + }, + { + "name": "skills.consume", + "durationMs": 84.441625, + "spanId": "a5a7081c6c2961ec", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + } + ] + }, + { + "label": "warm-17", + "elapsedMs": 257.58879100000195, + "nativeAssetMs": 24.06116699999984, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 839352320, + "cpuUserMs": 42.417, + "cpuSystemMs": 43.725, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "1a7b8b693b4c4e85be0773b6a3d522a5", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 91.829958, + "spanId": "0966be6de1c97482", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + }, + { + "name": "skills.rows", + "durationMs": 22.791625, + "spanId": "f1a2aa94c5da75b7", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.146834, + "spanId": "620f70dc51819b0d", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.523083, + "spanId": "8a041b6acb40ffd2", + "parentSpanId": "0f2acbbee58bcdef", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 37.086292, + "spanId": "0f2acbbee58bcdef", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + }, + { + "name": "skills.consume", + "durationMs": 46.412834, + "spanId": "b7d82ae57fc8b0da", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + } + ] + }, + { + "label": "warm-18", + "elapsedMs": 261.34670899999765, + "nativeAssetMs": 24.197124999998778, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 839811072, + "cpuUserMs": 41.926, + "cpuSystemMs": 47.502, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "161d8b331b6a5eb16c4703c7df25aa22", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.52425, + "spanId": "450dda07090d21e0", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + }, + { + "name": "skills.rows", + "durationMs": 24.316542, + "spanId": "cd3ff72582764824", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.990042, + "spanId": "ecb7218f0217a931", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.900792, + "spanId": "302f69462c259eb5", + "parentSpanId": "76d6fb44c9bd9169", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.195083, + "spanId": "76d6fb44c9bd9169", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + }, + { + "name": "skills.consume", + "durationMs": 48.113084, + "spanId": "fa7076037887129b", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + } + ] + }, + { + "label": "warm-19", + "elapsedMs": 255.9555419999997, + "nativeAssetMs": 22.198041999999987, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 840302592, + "cpuUserMs": 41.007, + "cpuSystemMs": 42.203, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "d6aa9695bfab2522a5b20285da1fb598", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 92.449917, + "spanId": "d46a9727b1ba1a04", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + }, + { + "name": "skills.rows", + "durationMs": 21.752667, + "spanId": "439112b181436b91", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.827208, + "spanId": "3aae300a50f483ec", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.7435, + "spanId": "30ccb2e21a60587b", + "parentSpanId": "5b0d15bafaa93022", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.64025, + "spanId": "5b0d15bafaa93022", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + }, + { + "name": "skills.consume", + "durationMs": 45.945459, + "spanId": "4f855ccc5cd02e6e", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + } + ] + }, + { + "label": "warm-20", + "elapsedMs": 302.01637499999924, + "nativeAssetMs": 55.15375000000131, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 840957952, + "cpuUserMs": 44.687, + "cpuSystemMs": 37.404, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "a65908be572847cf5d56c15739917e1e", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.346583, + "spanId": "b7578db1eec6e06b", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + }, + { + "name": "skills.rows", + "durationMs": 22.568333, + "spanId": "5cea64ceb62c4e03", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + }, + { + "name": "skills.version.read", + "durationMs": 24.110416, + "spanId": "f70e0dbbad2cd917", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + }, + { + "name": "skills.version.verify", + "durationMs": 21.076459, + "spanId": "7edc1debda754122", + "parentSpanId": "9834693c11018354", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 46.041083, + "spanId": "9834693c11018354", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + }, + { + "name": "skills.consume", + "durationMs": 79.26225, + "spanId": "0692bf561c7e0419", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + } + ] + }, + { + "label": "warm-new-service", + "elapsedMs": 405.37420900000143, + "nativeAssetMs": 59.024374999997235, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 841891840, + "cpuUserMs": 59.181, + "cpuSystemMs": 60.108, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "1d872a742343629c5e2a9550fce4e574", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 119.922875, + "spanId": "24ec4140ecbe6008", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + }, + { + "name": "skills.rows", + "durationMs": 31.560959, + "spanId": "f2247ca794d615c5", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + }, + { + "name": "skills.version.read", + "durationMs": 25.014542, + "spanId": "96aa7ea84f7a016a", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + }, + { + "name": "skills.version.verify", + "durationMs": 71.03075, + "spanId": "d10a2a22d516cfc4", + "parentSpanId": "4b5e1ed80d6d5e2b", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 108.683375, + "spanId": "4b5e1ed80d6d5e2b", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + }, + { + "name": "skills.consume", + "durationMs": 84.205833, + "spanId": "5da070654b83f17a", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + } + ] + } + ] + }, + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "46a4a05b61b399d18d1009085b6e54e9853563af98b3580318fb69290f0e05f1", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 10, + "instanceDisk": { + "allocatedBytes": 596471808, + "fileBytes": 485644240, + "files": 28572 + }, + "repositoryHash": "8905e1b5adbdbc1f4be27131dcc3065dddbe615f73e90f8e0ecd9a6349feab43", + "repositoryFiles": 1358, + "runtimeFiles": 1359, + "runtimeBytes": 22378235, + "samples": [ + { + "label": "cold", + "elapsedMs": 47645.3685, + "nativeAssetMs": 9308.106708, + "sqlResponseBytes": 29675, + "objectGets": 2, + "objectReadBytes": 13452324, + "rssBytes": 1166737408, + "cpuUserMs": 14069.023, + "cpuSystemMs": 17674.382, + "processPeakRssBytes": 1167867904, + "rootPoolQueries": 6, + "traceId": "562419a79efb8a11c006e311ca5593ad", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 114.230334, + "spanId": "8704b7082bbc332a", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.rows", + "durationMs": 28.7575, + "spanId": "8230d5d009e02885", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.125667, + "spanId": "50abf4ae64969ac3", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 2.812667, + "spanId": "5ccd1e0662caa241", + "parentSpanId": "076b8e8a5a95e497", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 3290.498, + "spanId": "076b8e8a5a95e497", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.060542, + "spanId": "e4c3915d46470f88", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.343125, + "spanId": "a246125338ced67b", + "parentSpanId": "3b97ef532785fd9b", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 2141.325542, + "spanId": "3b97ef532785fd9b", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.035167, + "spanId": "a2d82825e93112fe", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.268208, + "spanId": "0dbc0869fe1e7836", + "parentSpanId": "076ec67ef4c0e4b0", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 2875.915042, + "spanId": "076ec67ef4c0e4b0", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025917, + "spanId": "24db2198f4633e63", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.435417, + "spanId": "ce6761e421a0df42", + "parentSpanId": "f39908478d906933", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 3134.053334, + "spanId": "f39908478d906933", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024083, + "spanId": "907c1d86af17117d", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.440625, + "spanId": "a0ee9581c0f3865e", + "parentSpanId": "0de953dd6f306e20", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 3681.356417, + "spanId": "0de953dd6f306e20", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.038791, + "spanId": "93dd31171c070a8d", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.868, + "spanId": "c6614f7e35241394", + "parentSpanId": "421dc58c47a3c8b5", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 3857.3914999999997, + "spanId": "421dc58c47a3c8b5", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033833, + "spanId": "de20a361e93cc987", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 5.258875, + "spanId": "ea67abd8cae97096", + "parentSpanId": "f0afadbef89a51e1", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 8359.825959, + "spanId": "f0afadbef89a51e1", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033667, + "spanId": "4f3e15656690747c", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.254625, + "spanId": "e65c14f6a406806a", + "parentSpanId": "69f6cbb9eca241f5", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 5134.679292, + "spanId": "69f6cbb9eca241f5", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.045833, + "spanId": "b3c1b5f43355b889", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.758916, + "spanId": "c4f7a7232789b46e", + "parentSpanId": "b560806bec319f1c", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 4453.032834, + "spanId": "b560806bec319f1c", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019167, + "spanId": "0a7209360c8ac971", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.1945, + "spanId": "d97a270982776d3d", + "parentSpanId": "9a10a4cff81546bb", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 1135.038417, + "spanId": "9a10a4cff81546bb", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.consume", + "durationMs": 9338.957667, + "spanId": "e8119d7dca8e3b15", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + } + ] + }, + { + "label": "warm-1", + "elapsedMs": 745.369000000006, + "nativeAssetMs": 284.55312499999127, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170554880, + "cpuUserMs": 380.795, + "cpuSystemMs": 509.73, + "processPeakRssBytes": 1170554880, + "rootPoolQueries": 5, + "traceId": "75dd1e1c08d25c152d7d912a045c0fbe", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 107.59175, + "spanId": "8d3a42ad2215ac17", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.rows", + "durationMs": 51.915208, + "spanId": "c2817ffff0eb306f", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.039083, + "spanId": "10ef8db9b04a851f", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.432833, + "spanId": "32dae310ea0aefba", + "parentSpanId": "3593b113e12132c7", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.873917, + "spanId": "3593b113e12132c7", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022334, + "spanId": "6f169c4f2dc4aaad", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.416917, + "spanId": "bb7746d5304c6484", + "parentSpanId": "b84d63f3c526e369", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.334959, + "spanId": "b84d63f3c526e369", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.030958, + "spanId": "0e18fd8c95595f59", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.0795, + "spanId": "1252b0ee3c28cf2c", + "parentSpanId": "a4e8a4433c76223a", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.567958, + "spanId": "a4e8a4433c76223a", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018, + "spanId": "6c3df87cb2548efc", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.840208, + "spanId": "b8f70a96f8582eac", + "parentSpanId": "ac22c12c9a505540", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.007917, + "spanId": "ac22c12c9a505540", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02325, + "spanId": "0ac8bd8629948fe5", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.838959, + "spanId": "cf473ff3dffb73a7", + "parentSpanId": "09921bcadac73c2a", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.123042, + "spanId": "09921bcadac73c2a", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.037125, + "spanId": "9ac602ebdae9a7c0", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.50525, + "spanId": "91bf32a23e0aa45d", + "parentSpanId": "278ce350087a3829", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.624292, + "spanId": "278ce350087a3829", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025583, + "spanId": "4061a6e7cf923b66", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.344792, + "spanId": "ecad7c97ec014990", + "parentSpanId": "908e3e0338f7f70f", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.189708, + "spanId": "908e3e0338f7f70f", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031083, + "spanId": "677e298ae4de8e13", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.288875, + "spanId": "637a95854d822f44", + "parentSpanId": "0e676ca1b4dedd2a", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.309625, + "spanId": "0e676ca1b4dedd2a", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024959, + "spanId": "3166fa6d419f70bb", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 19.025417, + "spanId": "3073930e396c5dd6", + "parentSpanId": "6189f8652a95dbf8", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.079833, + "spanId": "6189f8652a95dbf8", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.041333, + "spanId": "350ae3c8654130ec", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.069166, + "spanId": "2bb2835f409c139b", + "parentSpanId": "62740e1ebd8a8b81", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.14625, + "spanId": "62740e1ebd8a8b81", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.consume", + "durationMs": 308.208708, + "spanId": "830437259e669f47", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + } + ] + }, + { + "label": "warm-2", + "elapsedMs": 688.4290000000037, + "nativeAssetMs": 270.29879100000835, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1173569536, + "cpuUserMs": 370.926, + "cpuSystemMs": 532.599, + "processPeakRssBytes": 1173569536, + "rootPoolQueries": 5, + "traceId": "f6fc891515b3863e44ee0f8be7d8e771", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 89.272625, + "spanId": "060902a01806a14a", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.rows", + "durationMs": 50.614, + "spanId": "14c8f817ee1fde6d", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.035792, + "spanId": "9a6bca85685eaf68", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.488291, + "spanId": "f21743da07c91d7d", + "parentSpanId": "26ffb0ad2b85e6e6", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.310958, + "spanId": "26ffb0ad2b85e6e6", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.030459, + "spanId": "6509859f300acbfe", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.33175, + "spanId": "85d229e0cb148197", + "parentSpanId": "7108fb34e6c3ed34", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.86625, + "spanId": "7108fb34e6c3ed34", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023458, + "spanId": "95824a1ef4a3d153", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.349791, + "spanId": "ba310aafbbb73679", + "parentSpanId": "b682295241226103", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.602417, + "spanId": "b682295241226103", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021958, + "spanId": "425fee7e6203d54f", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 27.363166, + "spanId": "68da68d25bed9811", + "parentSpanId": "54a57a0d129dafc0", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 29.389833, + "spanId": "54a57a0d129dafc0", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03025, + "spanId": "85bdb32ac10acaee", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.543167, + "spanId": "48a19633a4c0483c", + "parentSpanId": "9644b4750599e760", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.61325, + "spanId": "9644b4750599e760", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.027083, + "spanId": "fca1aa43d12c5efc", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.091917, + "spanId": "e446186f7cf4f100", + "parentSpanId": "394da3c5e72c46f9", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.961083, + "spanId": "394da3c5e72c46f9", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03475, + "spanId": "51d13872e8be4418", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.925833, + "spanId": "681d908abdba4307", + "parentSpanId": "05254d42369095cc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.240583, + "spanId": "05254d42369095cc", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.027166, + "spanId": "6f211ed57a63ccb2", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.896083, + "spanId": "7ec61f46ce93a098", + "parentSpanId": "8b73850164882936", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.643042, + "spanId": "8b73850164882936", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.027, + "spanId": "71f03873e2bf8778", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.00825, + "spanId": "e0f506a91f64172f", + "parentSpanId": "5fb74ddde9b42aa5", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.666084, + "spanId": "5fb74ddde9b42aa5", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03925, + "spanId": "2e1f399f54e054ac", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.569, + "spanId": "c99b61fed9306b2d", + "parentSpanId": "91fc34a65b258c3b", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.538708, + "spanId": "91fc34a65b258c3b", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.consume", + "durationMs": 294.15025, + "spanId": "f4b6905adaf92377", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + } + ] + }, + { + "label": "warm-3", + "elapsedMs": 711.9416250000068, + "nativeAssetMs": 304.56324999999197, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1177747456, + "cpuUserMs": 372.507, + "cpuSystemMs": 504.664, + "processPeakRssBytes": 1177747456, + "rootPoolQueries": 5, + "traceId": "6d46021c20cbd4e6e961d993ff35d86c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 92.585625, + "spanId": "18960143afde921b", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.rows", + "durationMs": 24.387084, + "spanId": "57283c99547dabf5", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.032042, + "spanId": "fdab68018c787042", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.407, + "spanId": "06fb60fe0fac4f25", + "parentSpanId": "0c6389049282a1ed", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.067708, + "spanId": "0c6389049282a1ed", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021084, + "spanId": "f78000633e8a81b6", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.873209, + "spanId": "7d42cecaf908469e", + "parentSpanId": "1f176d4f358fa349", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.892042, + "spanId": "1f176d4f358fa349", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019, + "spanId": "0d14b2dda9c298b4", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.823291, + "spanId": "cce261badbb534db", + "parentSpanId": "6c0d699a5db65238", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.001208, + "spanId": "6c0d699a5db65238", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.072917, + "spanId": "0b9fb3f688af352d", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.441083, + "spanId": "2262aa2633ffb70d", + "parentSpanId": "60c3903778bd4541", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.273167, + "spanId": "60c3903778bd4541", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025375, + "spanId": "300cf843ae793b5a", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.061167, + "spanId": "5ef267c4b94045a4", + "parentSpanId": "581f429299383960", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.309542, + "spanId": "581f429299383960", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029584, + "spanId": "68225fe3a8272bee", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.676583, + "spanId": "5ec4f509a77f98be", + "parentSpanId": "a05becc03594b286", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.906875, + "spanId": "a05becc03594b286", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.043125, + "spanId": "6c17e2bf970a441c", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.083916, + "spanId": "4024235e52c185b5", + "parentSpanId": "0783e9ba182ad2d0", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 24.032209, + "spanId": "0783e9ba182ad2d0", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.044958, + "spanId": "40468e633b3f0863", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.144042, + "spanId": "3da9d7442ed346cb", + "parentSpanId": "4def2c65a95dc749", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.094542, + "spanId": "4def2c65a95dc749", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022208, + "spanId": "ca52d5cd6617c596", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.482292, + "spanId": "5dea864f4e97a5bd", + "parentSpanId": "26736b05eccf8260", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.9115, + "spanId": "26736b05eccf8260", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.046458, + "spanId": "b10c3280c1d60f44", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.391084, + "spanId": "29fbdde04442cb25", + "parentSpanId": "e6c16853ff1170b4", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.140959, + "spanId": "e6c16853ff1170b4", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.consume", + "durationMs": 327.715667, + "spanId": "dd0c896936d72f4b", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + } + ] + }, + { + "label": "warm-4", + "elapsedMs": 815.8250839999964, + "nativeAssetMs": 304.6120410000003, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1178189824, + "cpuUserMs": 391.625, + "cpuSystemMs": 554.741, + "processPeakRssBytes": 1178206208, + "rootPoolQueries": 5, + "traceId": "9f20b6f800b57fec38cb344d21fc6153", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 133.318542, + "spanId": "4613fa5f002ac318", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.rows", + "durationMs": 24.012959, + "spanId": "c94be52fa3505c23", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.039167, + "spanId": "61201cee5b056ad9", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.792167, + "spanId": "3aa72d77da9e4cdd", + "parentSpanId": "65159ed5d64df778", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.082667, + "spanId": "65159ed5d64df778", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031083, + "spanId": "3211c92366cbf0a1", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 23.405875, + "spanId": "8e6059475d0482d7", + "parentSpanId": "a1132748a4ceb117", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 27.072459, + "spanId": "a1132748a4ceb117", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.058667, + "spanId": "92c62687a08462e9", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.676208, + "spanId": "ad85f09e49538705", + "parentSpanId": "5e67cd60deb21440", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.205291, + "spanId": "5e67cd60deb21440", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031042, + "spanId": "babb8a3db6e51a71", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.413625, + "spanId": "eaad9fb2d366f9aa", + "parentSpanId": "65703ae92dbc1cff", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.003167, + "spanId": "65703ae92dbc1cff", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.041542, + "spanId": "967e915cd17dd81d", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.915667, + "spanId": "3bd57edb555c2f41", + "parentSpanId": "183119b9c6084300", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.244125, + "spanId": "183119b9c6084300", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.118833, + "spanId": "ab5f3ae30a1e0e68", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.589625, + "spanId": "91188a119576e5cd", + "parentSpanId": "988c15842263db37", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.700084, + "spanId": "988c15842263db37", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.043375, + "spanId": "4f3195aa6b439ada", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.826, + "spanId": "58637d350cab74eb", + "parentSpanId": "328bc410f5bd8c5c", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.670583, + "spanId": "328bc410f5bd8c5c", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0405, + "spanId": "3dd12efbe06671aa", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.727375, + "spanId": "ef323da3c3a16d57", + "parentSpanId": "5dd55a875f863fc4", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.299959, + "spanId": "5dd55a875f863fc4", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025334, + "spanId": "9331c9844ec3ecfd", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.875792, + "spanId": "d4eaad4a82125e36", + "parentSpanId": "3581cf04e567d7fc", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 62.447167, + "spanId": "3581cf04e567d7fc", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028625, + "spanId": "7c76e19e11df45cf", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.835459, + "spanId": "c0efe14f3c6afbf7", + "parentSpanId": "9762da0400e05ef6", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.268292, + "spanId": "9762da0400e05ef6", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.consume", + "durationMs": 327.780125, + "spanId": "b91ba87a9478fb2f", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + } + ] + }, + { + "label": "warm-5", + "elapsedMs": 723.6927079999878, + "nativeAssetMs": 281.0947080000042, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1181990912, + "cpuUserMs": 371.021, + "cpuSystemMs": 539.923, + "processPeakRssBytes": 1181990912, + "rootPoolQueries": 5, + "traceId": "2189e97eb2b25393001ffed519205068", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 91.356083, + "spanId": "d15c060883555679", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.rows", + "durationMs": 21.800792, + "spanId": "de58aa701a1a619c", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031125, + "spanId": "82972b8a1db50ca5", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.21825, + "spanId": "2a6879872a3ebba6", + "parentSpanId": "2f2090a815077c05", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.742084, + "spanId": "2f2090a815077c05", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021041, + "spanId": "0736fd8a38b586a8", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.880417, + "spanId": "5b7a7f5ec737b78d", + "parentSpanId": "9e37370a22c0f17a", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.337583, + "spanId": "9e37370a22c0f17a", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022042, + "spanId": "36362b73f345d107", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.485167, + "spanId": "40fe22ac7ee3c075", + "parentSpanId": "6b868a7405716b11", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.354125, + "spanId": "6b868a7405716b11", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021125, + "spanId": "0f5f5c5ce05fa71f", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.407125, + "spanId": "fa5b05b245447520", + "parentSpanId": "744dc935455d52dd", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.375458, + "spanId": "744dc935455d52dd", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023584, + "spanId": "9e67c269ffae6f3d", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.638958, + "spanId": "df125923d7c7a5c0", + "parentSpanId": "35553ee6e7609a6a", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.991875, + "spanId": "35553ee6e7609a6a", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026125, + "spanId": "8798f66101f1f69c", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.115541, + "spanId": "ddbc71a9ec29078c", + "parentSpanId": "35ce1079b6e3aefa", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.474416, + "spanId": "35ce1079b6e3aefa", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.035792, + "spanId": "33372845e5d9b915", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.943584, + "spanId": "cf445f9035cc12fa", + "parentSpanId": "135454c61abc83f7", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.460875, + "spanId": "135454c61abc83f7", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023041, + "spanId": "6d6bc6913a8143f9", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.297667, + "spanId": "3bc578f94789f967", + "parentSpanId": "ff668cad47ddbefa", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 51.456708, + "spanId": "ff668cad47ddbefa", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028708, + "spanId": "4c094a7e7ab86430", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.726291, + "spanId": "fa8e31d3286b7330", + "parentSpanId": "616cd86536b9ce60", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 25.449167, + "spanId": "616cd86536b9ce60", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021667, + "spanId": "61e9fc8861ae6b57", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.343333, + "spanId": "d5ab008d36ef2600", + "parentSpanId": "02ac698a9f091c03", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.328041, + "spanId": "02ac698a9f091c03", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.consume", + "durationMs": 305.00275, + "spanId": "3ffa09be181363e1", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + } + ] + }, + { + "label": "warm-6", + "elapsedMs": 692.828874999992, + "nativeAssetMs": 298.1707499999902, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1185988608, + "cpuUserMs": 386.279, + "cpuSystemMs": 578.683, + "processPeakRssBytes": 1185988608, + "rootPoolQueries": 5, + "traceId": "b8234d9a319106e47acd03fbe548a2c7", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 94.715458, + "spanId": "aedc353c832ab3ab", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.rows", + "durationMs": 26.055334, + "spanId": "3f840a5f081fd28c", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.036166, + "spanId": "bc63d9f863f2359f", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.514542, + "spanId": "a9fde683037ebb86", + "parentSpanId": "832c37b87b14fd37", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.874292, + "spanId": "832c37b87b14fd37", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.035584, + "spanId": "d6c35cf9c58d99e4", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.130625, + "spanId": "79bffe885104fe59", + "parentSpanId": "4969ea6c07e47718", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.417875, + "spanId": "4969ea6c07e47718", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.051083, + "spanId": "13a8a683b0df6665", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.830375, + "spanId": "a028233a6771c2cd", + "parentSpanId": "ad6fe420926a494b", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.871208, + "spanId": "ad6fe420926a494b", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033958, + "spanId": "50696452911da230", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.58475, + "spanId": "6225b360b4bd611b", + "parentSpanId": "4a8892281b4ba2c2", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.851916, + "spanId": "4a8892281b4ba2c2", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024584, + "spanId": "1e5a5399ba790ee2", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.436708, + "spanId": "6bc08708fa7239f0", + "parentSpanId": "819c612b7985c660", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.753458, + "spanId": "819c612b7985c660", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023, + "spanId": "cd9e637a96b3dd86", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.652042, + "spanId": "778e4fd621b673ff", + "parentSpanId": "eb273bc455fb4601", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.898334, + "spanId": "eb273bc455fb4601", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025875, + "spanId": "556260aeaebcad0e", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.072375, + "spanId": "5137f757dccbcb6b", + "parentSpanId": "94e1f8753a5d56a6", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.372208, + "spanId": "94e1f8753a5d56a6", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029208, + "spanId": "cb57647f1825967b", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.132334, + "spanId": "c8db5698f4e3c096", + "parentSpanId": "f1a8f951083ca5d9", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.198208, + "spanId": "f1a8f951083ca5d9", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026833, + "spanId": "73856b6453b41419", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.94675, + "spanId": "d6516b0f78be05e2", + "parentSpanId": "18f0869166b06f40", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.020625, + "spanId": "18f0869166b06f40", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.050375, + "spanId": "8857d49eac48a844", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.131375, + "spanId": "76ac8ae85f39fc08", + "parentSpanId": "506cdd5913671263", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.399042, + "spanId": "506cdd5913671263", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.consume", + "durationMs": 322.229167, + "spanId": "69e972fed1225673", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + } + ] + }, + { + "label": "warm-7", + "elapsedMs": 875.4426669999957, + "nativeAssetMs": 329.81108299999323, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1190395904, + "cpuUserMs": 392.125, + "cpuSystemMs": 650.12, + "processPeakRssBytes": 1190395904, + "rootPoolQueries": 5, + "traceId": "316101c00f4a73e281c068b814da6ebd", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 137.793208, + "spanId": "8983a1b297665486", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.rows", + "durationMs": 22.361834, + "spanId": "255b1aa3f8ee4398", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.071333, + "spanId": "c13a041b2e1ed54b", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.208625, + "spanId": "bfac1ef3a94b24e6", + "parentSpanId": "e4048d0e21b29698", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.335167, + "spanId": "e4048d0e21b29698", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.045416, + "spanId": "c61f5837a8dabf08", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.453167, + "spanId": "605cef8d689817c3", + "parentSpanId": "ae7f6a09d5bf281f", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.260542, + "spanId": "ae7f6a09d5bf281f", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.059209, + "spanId": "6c6404b76db640ce", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 24.3275, + "spanId": "a94395f15ab2d0e1", + "parentSpanId": "1ed17d32bf9beddc", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.069042, + "spanId": "1ed17d32bf9beddc", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020959, + "spanId": "2e7977fbc49ba76d", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 21.746166, + "spanId": "102e58245df91309", + "parentSpanId": "ef3cc8312d60f1f7", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 27.900084, + "spanId": "ef3cc8312d60f1f7", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026167, + "spanId": "e41fca6bc42bb793", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 24.673917, + "spanId": "2b9246acf3b3e46f", + "parentSpanId": "933697aa0968395b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.340042, + "spanId": "933697aa0968395b", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023208, + "spanId": "f3653a3a4d284e72", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 22.358917, + "spanId": "31ff787813b70581", + "parentSpanId": "67dc6bda595b8422", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.957584, + "spanId": "67dc6bda595b8422", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021583, + "spanId": "6b9a7f0cb5f08533", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 54.438042, + "spanId": "c167d29b2f2de2a0", + "parentSpanId": "30880fcbc23cba70", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 56.022125, + "spanId": "30880fcbc23cba70", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.070667, + "spanId": "3cae659df3dcd424", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 22.319375, + "spanId": "881eabb383fe5ba6", + "parentSpanId": "8ca12c644f1a168e", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.011667, + "spanId": "8ca12c644f1a168e", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028334, + "spanId": "41f55793e581ce51", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 25.668334, + "spanId": "fb162841b6c3b16a", + "parentSpanId": "39c5e0ab95d9cc43", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 27.801583, + "spanId": "39c5e0ab95d9cc43", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023875, + "spanId": "c29cee0e1be319e0", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 18.3515, + "spanId": "7d719525f6b74a9b", + "parentSpanId": "ae75a5763a3b8e71", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.310375, + "spanId": "ae75a5763a3b8e71", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.consume", + "durationMs": 354.05125, + "spanId": "90b6e830cb412dd4", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + } + ] + }, + { + "label": "warm-8", + "elapsedMs": 747.7878749999945, + "nativeAssetMs": 311.6350000000093, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1194590208, + "cpuUserMs": 377.021, + "cpuSystemMs": 801.387, + "processPeakRssBytes": 1194590208, + "rootPoolQueries": 5, + "traceId": "d97af7507b406a230f62012a303458d7", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.059, + "spanId": "e85ade283b58c3b6", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.rows", + "durationMs": 24.219583, + "spanId": "5b2a005ec4e6cc96", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.041125, + "spanId": "e91cfcfe26d4e206", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.435583, + "spanId": "ee1fa8de62f56baf", + "parentSpanId": "3511b3eeb3dba368", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.081292, + "spanId": "3511b3eeb3dba368", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019917, + "spanId": "0a7333292be748e4", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 18.895333, + "spanId": "66e0a5dac460e031", + "parentSpanId": "777b5794cb874b00", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 21.181709, + "spanId": "777b5794cb874b00", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018542, + "spanId": "d287c125aaaa6558", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.322875, + "spanId": "8bc9f5a3e7d319b0", + "parentSpanId": "cd8ab9ee8b47f0f0", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.875666, + "spanId": "cd8ab9ee8b47f0f0", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024959, + "spanId": "de095afc10fd80b2", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 19.542625, + "spanId": "e37758adb781fc96", + "parentSpanId": "cb15d30f17065a99", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 24.514459, + "spanId": "cb15d30f17065a99", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02375, + "spanId": "e21195dbdb2f475e", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.529583, + "spanId": "35032f3b8e2bece8", + "parentSpanId": "e461e99d3adb926b", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.925333, + "spanId": "e461e99d3adb926b", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019958, + "spanId": "a91cb11f65236054", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 18.435166, + "spanId": "8b4700bb388385d1", + "parentSpanId": "cb9e85e1397e3229", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.58525, + "spanId": "cb9e85e1397e3229", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031417, + "spanId": "e92616688b6c47ad", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 23.013458, + "spanId": "0a83e6a40c90f355", + "parentSpanId": "d05763dc69025db8", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 24.870125, + "spanId": "d05763dc69025db8", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017834, + "spanId": "6c79dd54c0457ac1", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 19.390833, + "spanId": "5e5d700631a4371a", + "parentSpanId": "7f024da6b0de0e17", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 21.392167, + "spanId": "7f024da6b0de0e17", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029667, + "spanId": "e5d5d425e699fa24", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.841541, + "spanId": "88b381ac1b85e212", + "parentSpanId": "f07c0ba0524ed071", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.474833, + "spanId": "f07c0ba0524ed071", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018875, + "spanId": "96573807066ec166", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.333583, + "spanId": "c2a35f6b9180ab40", + "parentSpanId": "f435eb1091231cb6", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.223542, + "spanId": "f435eb1091231cb6", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.consume", + "durationMs": 334.012625, + "spanId": "ae008dbf63f70329", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + } + ] + }, + { + "label": "warm-9", + "elapsedMs": 671.4019580000022, + "nativeAssetMs": 283.660916000008, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196359680, + "cpuUserMs": 380.548, + "cpuSystemMs": 663.329, + "processPeakRssBytes": 1196359680, + "rootPoolQueries": 5, + "traceId": "158c21552f85208b7e9205746855cdd1", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.567416, + "spanId": "0a154252dd392df2", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.rows", + "durationMs": 24.245459, + "spanId": "b9bdfe482150bf1f", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.037667, + "spanId": "a9cba88446216f7b", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.801667, + "spanId": "581512d6a85f27bc", + "parentSpanId": "155c9905c1352bf3", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.220375, + "spanId": "155c9905c1352bf3", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029042, + "spanId": "f6afce507e8571df", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.729458, + "spanId": "d5261044154b7f4d", + "parentSpanId": "ad653bb27ffafca4", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 21.094083, + "spanId": "ad653bb27ffafca4", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020416, + "spanId": "9f9623ba8226000f", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.672042, + "spanId": "719edc30dd8156dd", + "parentSpanId": "ad1ad65946448006", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.543625, + "spanId": "ad1ad65946448006", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029666, + "spanId": "073c115f656e8b77", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.733041, + "spanId": "e4f0b206dcfa056a", + "parentSpanId": "3f3d03c21cd5ca86", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.21625, + "spanId": "3f3d03c21cd5ca86", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0215, + "spanId": "ff107a15d448b0a3", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.275625, + "spanId": "d4b4122de2e1053e", + "parentSpanId": "c996382893a4d33d", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.548458, + "spanId": "c996382893a4d33d", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017083, + "spanId": "b8c87b25b2c138a9", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.757958, + "spanId": "40931943da3fdf4d", + "parentSpanId": "e05ec6ad3f3d09df", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.661125, + "spanId": "e05ec6ad3f3d09df", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017167, + "spanId": "0c0a60e224ebbc29", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.045417, + "spanId": "f1905953fba53348", + "parentSpanId": "f8823bc0909432a9", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.995375, + "spanId": "f8823bc0909432a9", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022625, + "spanId": "c0b314908ce4314f", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.8535, + "spanId": "6476499c1378babc", + "parentSpanId": "a169be093b96c12d", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.802417, + "spanId": "a169be093b96c12d", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016792, + "spanId": "c00e6fa26d4e477e", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.57775, + "spanId": "ce3e66f87abcd5af", + "parentSpanId": "28766b527b03d680", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.794083, + "spanId": "28766b527b03d680", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0185, + "spanId": "1725313dbc38cb3f", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.994292, + "spanId": "d1fe27e6f0fb88c2", + "parentSpanId": "852b533fc3d1c41d", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.337375, + "spanId": "852b533fc3d1c41d", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.consume", + "durationMs": 307.7825, + "spanId": "525e896569b02a35", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + } + ] + }, + { + "label": "warm-10", + "elapsedMs": 668.2096250000031, + "nativeAssetMs": 277.69979199999943, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196769280, + "cpuUserMs": 371.548, + "cpuSystemMs": 562.298, + "processPeakRssBytes": 1196769280, + "rootPoolQueries": 5, + "traceId": "9c3c4b127e6fbf0fcbda0a3291463287", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 98.237833, + "spanId": "fb7a743232ce9d84", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.rows", + "durationMs": 23.887291, + "spanId": "003ef8acec63e9aa", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.040625, + "spanId": "d19c5395c411bb1c", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.5185, + "spanId": "c587d889916fef44", + "parentSpanId": "1422e5cd1276f48b", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.843208, + "spanId": "1422e5cd1276f48b", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01825, + "spanId": "d710fded7e41266e", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.854125, + "spanId": "f3952231be884e32", + "parentSpanId": "405056cd329b2d1e", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.708834, + "spanId": "405056cd329b2d1e", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017458, + "spanId": "6fe40fb9ccbb0c49", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.518834, + "spanId": "0d90499ee67ac369", + "parentSpanId": "11139fe4d944c2d9", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 27.902875, + "spanId": "11139fe4d944c2d9", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017875, + "spanId": "aeef75b4a3396dc6", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.197625, + "spanId": "20daa90f77d2a9ea", + "parentSpanId": "1b7a9a50b414387a", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.225542, + "spanId": "1b7a9a50b414387a", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018708, + "spanId": "5a799fe73cff2033", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.3595, + "spanId": "337b84d6c76fb414", + "parentSpanId": "879f84c961c65633", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.210291, + "spanId": "879f84c961c65633", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.047541, + "spanId": "7650913d055ec95d", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.410417, + "spanId": "c9fa4a4600d02fc3", + "parentSpanId": "470a4a902b7de434", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.467208, + "spanId": "470a4a902b7de434", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023291, + "spanId": "925c4d46895a8e09", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.145084, + "spanId": "fcafe69563f6ea82", + "parentSpanId": "2e2bcbda55e1ef79", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.202416, + "spanId": "2e2bcbda55e1ef79", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021833, + "spanId": "1b7ef7a7c0a09c11", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.990958, + "spanId": "a8913945898c0046", + "parentSpanId": "ad78fdeffab5b633", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.396667, + "spanId": "ad78fdeffab5b633", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0325, + "spanId": "5691e4ea778dec6d", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.330709, + "spanId": "7186a056af575152", + "parentSpanId": "0c5cbfe75b1be084", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.621417, + "spanId": "0c5cbfe75b1be084", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025792, + "spanId": "f0538f6403c45779", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.0955, + "spanId": "836cadcb20dbc05b", + "parentSpanId": "a8f980e4360b04da", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.268583, + "spanId": "a8f980e4360b04da", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.consume", + "durationMs": 301.668459, + "spanId": "49f584232f6a960b", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + } + ] + }, + { + "label": "warm-11", + "elapsedMs": 672.346458, + "nativeAssetMs": 285.53370800000266, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196769280, + "cpuUserMs": 358.553, + "cpuSystemMs": 552.457, + "processPeakRssBytes": 1196769280, + "rootPoolQueries": 5, + "traceId": "022c6b37377415db1545b3c051352546", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 94.570666, + "spanId": "cc56babdbd788833", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.rows", + "durationMs": 23.555584, + "spanId": "909eb1818b1eb542", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.122708, + "spanId": "f57ecdf5b3ee7f92", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.567625, + "spanId": "5eeee09f44a72910", + "parentSpanId": "f46fb8b9a3ab5bf8", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 40.547333, + "spanId": "f46fb8b9a3ab5bf8", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017, + "spanId": "48330e3aa4907abd", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.450166, + "spanId": "74b594fcc52334b0", + "parentSpanId": "4546edcd47c7d4cd", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.484375, + "spanId": "4546edcd47c7d4cd", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01925, + "spanId": "7cffb65bf3732f88", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.773791, + "spanId": "7ca38aea3f39c353", + "parentSpanId": "838e542666b6ea70", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.954625, + "spanId": "838e542666b6ea70", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026625, + "spanId": "8f8d6365132c6c79", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.178542, + "spanId": "0736007e870ff3f6", + "parentSpanId": "3b20728e82bd1475", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.188125, + "spanId": "3b20728e82bd1475", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026292, + "spanId": "40f790790e67a847", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.658959, + "spanId": "848108ce9b39120c", + "parentSpanId": "3a7d0edecbd2ebc2", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.386625, + "spanId": "3a7d0edecbd2ebc2", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016084, + "spanId": "88c0952332d27798", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.141416, + "spanId": "8823f050996424d0", + "parentSpanId": "b8363f9a96aa799a", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.486459, + "spanId": "b8363f9a96aa799a", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0135, + "spanId": "33abe9d71139acf5", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.473084, + "spanId": "f0126683d42010da", + "parentSpanId": "dff8e4078e5d0680", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.226791, + "spanId": "dff8e4078e5d0680", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013208, + "spanId": "34af429e3bd5af36", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.648833, + "spanId": "9134dfd32cae1c90", + "parentSpanId": "e520ad331c98aae6", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.264625, + "spanId": "e520ad331c98aae6", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020792, + "spanId": "c9cbd05cda0f7176", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.428833, + "spanId": "8440bc51427b378f", + "parentSpanId": "9e29c65d09b830ed", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.277541, + "spanId": "9e29c65d09b830ed", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021, + "spanId": "fd25067abff5ee02", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.303125, + "spanId": "2a43ff36181534cd", + "parentSpanId": "bb6c030e8ae7b6e2", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.08825, + "spanId": "bb6c030e8ae7b6e2", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.consume", + "durationMs": 309.249375, + "spanId": "27139f6dc36c1c0a", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + } + ] + }, + { + "label": "warm-12", + "elapsedMs": 723.1867499999935, + "nativeAssetMs": 311.0507079999952, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196785664, + "cpuUserMs": 387.377, + "cpuSystemMs": 567.72, + "processPeakRssBytes": 1196785664, + "rootPoolQueries": 5, + "traceId": "8dfbffb029157b820792369a471e6950", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 95.063042, + "spanId": "3637e24cc45eb6f8", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.rows", + "durationMs": 24.186, + "spanId": "04c5e857670a6577", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031833, + "spanId": "e2c84cfbdafc16be", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 25.678625, + "spanId": "ea1ec70180d9e2be", + "parentSpanId": "5b6e2ac5c61f4192", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 31.75025, + "spanId": "5b6e2ac5c61f4192", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03225, + "spanId": "64447a9d7423c531", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.461708, + "spanId": "886ddcb08ceb6d24", + "parentSpanId": "959da2cb8987c28f", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.344541, + "spanId": "959da2cb8987c28f", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0155, + "spanId": "63e8574e224b555a", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.926041, + "spanId": "59e409a06edfefc5", + "parentSpanId": "4679b8efd0317998", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.536375, + "spanId": "4679b8efd0317998", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02025, + "spanId": "6df0eec22ff2a65e", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.600417, + "spanId": "a7fb66554d618bc9", + "parentSpanId": "0acc11ae45d6e766", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.112916, + "spanId": "0acc11ae45d6e766", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025375, + "spanId": "9b55c3e3c28c309a", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.815417, + "spanId": "3d6a2a276d37aa9c", + "parentSpanId": "ad326efafecb2727", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.220583, + "spanId": "ad326efafecb2727", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019959, + "spanId": "8b2e8b9220704cf0", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.458833, + "spanId": "2584e7e442577968", + "parentSpanId": "844781a1b3f9415e", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.14875, + "spanId": "844781a1b3f9415e", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023833, + "spanId": "ea257174968f741a", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.802917, + "spanId": "9a03a17253daa3ed", + "parentSpanId": "f687dd834f6079db", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.741791, + "spanId": "f687dd834f6079db", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020167, + "spanId": "e6b8a4564d342577", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.60725, + "spanId": "0c643f6f7f01f583", + "parentSpanId": "76da6221cbae5abb", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.825, + "spanId": "76da6221cbae5abb", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.070375, + "spanId": "d03955486894a051", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.219666, + "spanId": "af793ebcf23992b6", + "parentSpanId": "d0c49f702252f68c", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.298292, + "spanId": "d0c49f702252f68c", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015959, + "spanId": "57ee0f4ae2906905", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.218291, + "spanId": "e93b5674b3f692dd", + "parentSpanId": "ffc2b16ae38cc872", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.2255, + "spanId": "ffc2b16ae38cc872", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.consume", + "durationMs": 334.045792, + "spanId": "5674cdb81ce53f15", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + } + ] + }, + { + "label": "warm-13", + "elapsedMs": 792.8505829999922, + "nativeAssetMs": 345.88183300000674, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196851200, + "cpuUserMs": 398.725, + "cpuSystemMs": 540.204, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "ab0e83463bc450d0d50b4c69ad1e11e9", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 96.953, + "spanId": "7c071cb28e7358dd", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.rows", + "durationMs": 24.012458, + "spanId": "55bc10a5898a0f63", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033459, + "spanId": "1b30fc4efa3c514e", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 19.479875, + "spanId": "4dacde57a9f4ffa6", + "parentSpanId": "5764f9f070fd5427", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.039208, + "spanId": "5764f9f070fd5427", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020791, + "spanId": "96bd7c924d37a4d5", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.505, + "spanId": "e20a0ca38ae05860", + "parentSpanId": "6d88788e06d2cded", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 29.035, + "spanId": "6d88788e06d2cded", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019667, + "spanId": "d38a245da09258ae", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.77, + "spanId": "710819766b964ec0", + "parentSpanId": "f846e8a5a57c6163", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.220208, + "spanId": "f846e8a5a57c6163", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.089, + "spanId": "0b85226d0792714e", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.778542, + "spanId": "d827795876f25b55", + "parentSpanId": "9475b0ec08321af7", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 30.210333, + "spanId": "9475b0ec08321af7", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.027417, + "spanId": "4f9d9e62f4dc4648", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.429, + "spanId": "5d9e36db871ea0bd", + "parentSpanId": "872d044bf3336b3d", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.475542, + "spanId": "872d044bf3336b3d", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.042167, + "spanId": "5c38a59db88266cb", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 18.609291, + "spanId": "233b6d2849326f87", + "parentSpanId": "3620704c0d6902df", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 21.441583, + "spanId": "3620704c0d6902df", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.037083, + "spanId": "db7d1d6aa1bbac5e", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.889792, + "spanId": "7bbfc1b7443b3d42", + "parentSpanId": "4e59c150960b43e5", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.036917, + "spanId": "4e59c150960b43e5", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018, + "spanId": "0cda4055390cd6df", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.488583, + "spanId": "cc4f3e43e6bff564", + "parentSpanId": "e59fdf7193dec64d", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.103583, + "spanId": "e59fdf7193dec64d", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019333, + "spanId": "a8edd2ed1679a177", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.51625, + "spanId": "09d369cb0890c7bb", + "parentSpanId": "2f86ee841ee523f4", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.863791, + "spanId": "2f86ee841ee523f4", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020416, + "spanId": "b85bb2aa450ba319", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.116333, + "spanId": "a603978033894295", + "parentSpanId": "ae2c40acf31050f3", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.185, + "spanId": "ae2c40acf31050f3", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.consume", + "durationMs": 370.029, + "spanId": "aeffbdcade2d443a", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + } + ] + }, + { + "label": "warm-14", + "elapsedMs": 752.5533330000035, + "nativeAssetMs": 293.2099579999922, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1169276928, + "cpuUserMs": 380.094, + "cpuSystemMs": 604.486, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "c19c6ba80aece473d3eca1071182f6af", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 96.714458, + "spanId": "a505ecd060a814f5", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.rows", + "durationMs": 23.460459, + "spanId": "cbe82a934258a363", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.034833, + "spanId": "1d7a20b0d88dfa07", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.483459, + "spanId": "c9386cc08189b5b2", + "parentSpanId": "06eb9689f0428064", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.7495, + "spanId": "06eb9689f0428064", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017834, + "spanId": "ca5fc3a38e3d5f6a", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.10375, + "spanId": "8e19020c784ebcc3", + "parentSpanId": "33acfedfb80c46cb", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.613917, + "spanId": "33acfedfb80c46cb", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017084, + "spanId": "4c08eedb3a58aa33", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.501209, + "spanId": "25e06bf2c14e5eff", + "parentSpanId": "95dfc2a9e497f11a", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.521375, + "spanId": "95dfc2a9e497f11a", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017791, + "spanId": "5d3c3963206290f4", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.547459, + "spanId": "cb352b722199d5e2", + "parentSpanId": "354de89589f230c5", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.6525, + "spanId": "354de89589f230c5", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019375, + "spanId": "bd4fa4aa651a056e", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.349834, + "spanId": "d2b551d170b127b1", + "parentSpanId": "4139293c0a84d3a6", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.434083, + "spanId": "4139293c0a84d3a6", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016958, + "spanId": "f0ab419d2e2a0664", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.505833, + "spanId": "2c7a04272875220c", + "parentSpanId": "c5f742ee063de31b", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.832792, + "spanId": "c5f742ee063de31b", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017584, + "spanId": "0089a417ef6549d1", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.226334, + "spanId": "52867b3fb96547ae", + "parentSpanId": "4c0ffb3a3e1178bd", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.4505, + "spanId": "4c0ffb3a3e1178bd", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024084, + "spanId": "2bb307893398218b", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 21.191625, + "spanId": "06a235e04a8cd431", + "parentSpanId": "9e393715b4f2cd74", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 69.550542, + "spanId": "9e393715b4f2cd74", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019541, + "spanId": "de389ef30eb65ef2", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.9225, + "spanId": "5c0b58d67ea27a19", + "parentSpanId": "2196e41e087c3f93", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.361041, + "spanId": "2196e41e087c3f93", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018416, + "spanId": "bc5f20d4507a68ed", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.256625, + "spanId": "8b923960f779829f", + "parentSpanId": "68018e98e4297be6", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.004375, + "spanId": "68018e98e4297be6", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.consume", + "durationMs": 317.462292, + "spanId": "5e7be438767e9c5f", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + } + ] + }, + { + "label": "warm-15", + "elapsedMs": 678.2098330000008, + "nativeAssetMs": 292.1170829999901, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170112512, + "cpuUserMs": 368.551, + "cpuSystemMs": 601.134, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "3b93d864814cb1d637c2a3ee4fc8f7ee", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 95.106333, + "spanId": "f0ed73d3050e9260", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.rows", + "durationMs": 22.830958, + "spanId": "e9159aff074939a1", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.030667, + "spanId": "39b0cfaab1ecbb00", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.606041, + "spanId": "628ca4f20dee0e3f", + "parentSpanId": "08e5af25f48d60c9", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.862834, + "spanId": "08e5af25f48d60c9", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017333, + "spanId": "0b0ef4b89eab4f41", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.524166, + "spanId": "e4ce400512994b1f", + "parentSpanId": "8ea7d402824d6a95", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.716958, + "spanId": "8ea7d402824d6a95", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.036041, + "spanId": "17a9b85b1d219230", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.863583, + "spanId": "37898c03b730c413", + "parentSpanId": "b13c92d9e6ae479f", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.674708, + "spanId": "b13c92d9e6ae479f", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018875, + "spanId": "3b5869683f013175", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.112709, + "spanId": "4df180f3ac88ac05", + "parentSpanId": "d1b465639e6d2299", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.175416, + "spanId": "d1b465639e6d2299", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017083, + "spanId": "8dcbe861fd91a538", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.086042, + "spanId": "a2a2bba6e9a29ee5", + "parentSpanId": "48781e504377a7f0", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.219375, + "spanId": "48781e504377a7f0", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017541, + "spanId": "25509da5b6495d0d", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.891375, + "spanId": "f5bfd7f7bf8f1d2b", + "parentSpanId": "db6850b5ccce8550", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.721125, + "spanId": "db6850b5ccce8550", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033458, + "spanId": "f87ebc9e1f5ec255", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.967584, + "spanId": "d2b6db2b3362ee72", + "parentSpanId": "6ff618f2d8a2af30", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.916875, + "spanId": "6ff618f2d8a2af30", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022792, + "spanId": "12940b6b35524eb4", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.824041, + "spanId": "3b988ad31631a66a", + "parentSpanId": "e29bdd7f785ab7da", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.072959, + "spanId": "e29bdd7f785ab7da", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022167, + "spanId": "095c039d99fb0b49", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.191459, + "spanId": "44fd7ada1602c23a", + "parentSpanId": "35ab9a95079410c0", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.235167, + "spanId": "35ab9a95079410c0", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02175, + "spanId": "167d297d6daec949", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.836541, + "spanId": "65563c8a32a1ee58", + "parentSpanId": "6dc5d5f206c06612", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.779041, + "spanId": "6dc5d5f206c06612", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.consume", + "durationMs": 314.572792, + "spanId": "8ff40b67f4d4f910", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + } + ] + }, + { + "label": "warm-16", + "elapsedMs": 666.0508750000008, + "nativeAssetMs": 289.2593749999942, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170112512, + "cpuUserMs": 352.025, + "cpuSystemMs": 574.008, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "d227d3ecd0965bb710050923c10cb8fc", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 89.368875, + "spanId": "5f3f0bdfb48cbef9", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.rows", + "durationMs": 21.964416, + "spanId": "eeb2387367889517", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031917, + "spanId": "27fd3661a847fbd0", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.822791, + "spanId": "726648458f75d9b7", + "parentSpanId": "6766f1d0fed61f82", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.922, + "spanId": "6766f1d0fed61f82", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.05275, + "spanId": "9f5816fba6b55d4d", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.012792, + "spanId": "f847ef4c717eccb4", + "parentSpanId": "42986c651cbe40e8", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 28.399708, + "spanId": "42986c651cbe40e8", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016375, + "spanId": "242d44665082fbb7", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.813042, + "spanId": "7461be53148cfb26", + "parentSpanId": "8c196cb1b2b5f9ee", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.390708, + "spanId": "8c196cb1b2b5f9ee", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016167, + "spanId": "26061f4b061b03ed", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.804459, + "spanId": "5ec473613d1b210a", + "parentSpanId": "f13db5269b06f060", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.613958, + "spanId": "f13db5269b06f060", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.039, + "spanId": "2525e86ec6b917be", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.988917, + "spanId": "66ce3f7a716e9c5b", + "parentSpanId": "03cf963c27ba5e1b", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.177667, + "spanId": "03cf963c27ba5e1b", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01825, + "spanId": "04dd3c050cb57af1", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.381291, + "spanId": "dadc901e937b3c08", + "parentSpanId": "5faa408d4ba3ee0a", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.156125, + "spanId": "5faa408d4ba3ee0a", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015125, + "spanId": "d9f8803fb8ecaf1b", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.229, + "spanId": "552e76c6ae1af094", + "parentSpanId": "f967a1b6eb4ad0a9", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.78275, + "spanId": "f967a1b6eb4ad0a9", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018583, + "spanId": "8a3eee52f009a428", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.269917, + "spanId": "8cdc7e8455100713", + "parentSpanId": "0675393ed1dfa077", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.112542, + "spanId": "0675393ed1dfa077", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01575, + "spanId": "f8d579620d5dcb52", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.170042, + "spanId": "3c1caeca4b857bc7", + "parentSpanId": "f1fbce3d84b6b93f", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.001084, + "spanId": "f1fbce3d84b6b93f", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03325, + "spanId": "bdcd4d658d256acb", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.165125, + "spanId": "2f0d46b5e557e7c2", + "parentSpanId": "8a74b68966637939", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.528375, + "spanId": "8a74b68966637939", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.consume", + "durationMs": 310.610583, + "spanId": "2eaff2f06a8504b8", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + } + ] + }, + { + "label": "warm-17", + "elapsedMs": 629.3935419999907, + "nativeAssetMs": 265.11266699999396, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170178048, + "cpuUserMs": 342.554, + "cpuSystemMs": 539.634, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "0959ca2eb377863360c991b424da0f13", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.912875, + "spanId": "71c9a03dafcde650", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.rows", + "durationMs": 22.80825, + "spanId": "803a3f22bf442f71", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0285, + "spanId": "ebb5e4a7e13b1369", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.10625, + "spanId": "fef12f81d694e925", + "parentSpanId": "e3e411a49b9a6d4e", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.957208, + "spanId": "e3e411a49b9a6d4e", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03675, + "spanId": "57a4eafff7f48155", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.771167, + "spanId": "f36ee51d0d65dfcc", + "parentSpanId": "a9bfba0d0c1914c4", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.234625, + "spanId": "a9bfba0d0c1914c4", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.043125, + "spanId": "b86350c4b6e2c429", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.41425, + "spanId": "4ed5ea96531cd339", + "parentSpanId": "02d32218dcf1d2e5", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.883125, + "spanId": "02d32218dcf1d2e5", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020625, + "spanId": "bf2b3b7520d11797", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.661042, + "spanId": "f70f9849e221f567", + "parentSpanId": "510fef9f53405cb5", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.160041, + "spanId": "510fef9f53405cb5", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01325, + "spanId": "aba98a7709122283", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.827708, + "spanId": "baa65af8721c1b92", + "parentSpanId": "db31fcfc11e8307f", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 24.692791, + "spanId": "db31fcfc11e8307f", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015208, + "spanId": "f65a29890fe31116", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.929291, + "spanId": "e812810129fc4914", + "parentSpanId": "ebc6c09b543bdef1", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.221125, + "spanId": "ebc6c09b543bdef1", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013541, + "spanId": "85eb3477665173f2", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.543084, + "spanId": "5cdf9cf3c2101660", + "parentSpanId": "8cf973386c605384", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.558708, + "spanId": "8cf973386c605384", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.040166, + "spanId": "827b53fcc7d60a7e", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.980125, + "spanId": "76b1056aaf7238a5", + "parentSpanId": "6ccb285228518d06", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.877833, + "spanId": "6ccb285228518d06", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.043459, + "spanId": "adb39cd953ef889d", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.473333, + "spanId": "a3a54e035cc038f2", + "parentSpanId": "d91a62ca7d452bf3", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.260083, + "spanId": "d91a62ca7d452bf3", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028625, + "spanId": "4785cf6f969071a8", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.676208, + "spanId": "384b2b4d06ee9288", + "parentSpanId": "6cb6f53c6c887684", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.268417, + "spanId": "6cb6f53c6c887684", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.consume", + "durationMs": 287.488583, + "spanId": "230e657720325807", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + } + ] + }, + { + "label": "warm-18", + "elapsedMs": 587.9979170000006, + "nativeAssetMs": 266.7930420000048, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170194432, + "cpuUserMs": 323.898, + "cpuSystemMs": 502.382, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "7add063fd46cd3f784136c6c1774c933", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 85.881917, + "spanId": "17523936b48291e6", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.rows", + "durationMs": 23.638917, + "spanId": "efbebe2b7783a4c9", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025292, + "spanId": "3cfb8377a8ed5149", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.930458, + "spanId": "593ac73ea679a6d4", + "parentSpanId": "52119f530cafe0e8", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.315167, + "spanId": "52119f530cafe0e8", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014792, + "spanId": "59225536d3d6cbe4", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.476459, + "spanId": "c126080b47be7724", + "parentSpanId": "94b7343b63d76f45", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.521333, + "spanId": "94b7343b63d76f45", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.012458, + "spanId": "272cc0bd62f9fd1b", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.334125, + "spanId": "c5a3a1caf17e4f53", + "parentSpanId": "4708c01925f1e494", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.398791, + "spanId": "4708c01925f1e494", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.012083, + "spanId": "99c4295c2da6c50d", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.18875, + "spanId": "ebf451a41a0c342a", + "parentSpanId": "5af925b154caaf30", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.28725, + "spanId": "5af925b154caaf30", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013792, + "spanId": "39bb8dd7233e6de7", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.638792, + "spanId": "816b0c3449805d30", + "parentSpanId": "446e46ba9ff2a2fe", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.516792, + "spanId": "446e46ba9ff2a2fe", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014625, + "spanId": "501c1fcf0c0ca3f4", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.474625, + "spanId": "32eb967849144dda", + "parentSpanId": "83c56970fac108ca", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 9.49525, + "spanId": "83c56970fac108ca", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013, + "spanId": "ccfeeb6e5c0d5d2f", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.691084, + "spanId": "1d1ff3fc2c7350b5", + "parentSpanId": "3094093d4bf3570e", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.010166, + "spanId": "3094093d4bf3570e", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013417, + "spanId": "f645cff1e9e7b7bf", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.088333, + "spanId": "92a8a3ae1618daeb", + "parentSpanId": "2f65e778eac62cb5", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.200708, + "spanId": "2f65e778eac62cb5", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02975, + "spanId": "c10bac3df4223c75", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.962708, + "spanId": "7ec439ede98d8b07", + "parentSpanId": "792562ab10e88b07", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.095375, + "spanId": "792562ab10e88b07", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015459, + "spanId": "b8797e7a20aa91be", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.073875, + "spanId": "397691258e7ada02", + "parentSpanId": "fe1c5ed3b4b8b259", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.463334, + "spanId": "fe1c5ed3b4b8b259", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.consume", + "durationMs": 289.033667, + "spanId": "f9aafc385d9a93e8", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + } + ] + }, + { + "label": "warm-19", + "elapsedMs": 559.9573340000061, + "nativeAssetMs": 234.43945900000108, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170194432, + "cpuUserMs": 295.048, + "cpuSystemMs": 492.825, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "369dbe7d6a967e11de29815c49a8ad5c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 86.37125, + "spanId": "f88b9f8385f8dda6", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.rows", + "durationMs": 23.303958, + "spanId": "a399c47cacc9cc9e", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024958, + "spanId": "ce3f7468d95e92d0", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.060584, + "spanId": "ad1b99c3023c526b", + "parentSpanId": "7debd6f6cb31e3ba", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.376916, + "spanId": "7debd6f6cb31e3ba", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01775, + "spanId": "d873cf09b3d101bb", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.433333, + "spanId": "7c4d5c3b3ddab135", + "parentSpanId": "3d3da8cb9b92ebcb", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.509167, + "spanId": "3d3da8cb9b92ebcb", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013125, + "spanId": "3b55d90f6b3e02ed", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.914791, + "spanId": "df0b9fa1de262d09", + "parentSpanId": "da65185486325019", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 9.796666, + "spanId": "da65185486325019", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013667, + "spanId": "b06b8f24383889bb", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.97275, + "spanId": "ce3f59e14d660abb", + "parentSpanId": "62055cb41f4c3f9d", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.699666, + "spanId": "62055cb41f4c3f9d", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028041, + "spanId": "ca4e6075f2c156fe", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.249209, + "spanId": "d3550ddb45490add", + "parentSpanId": "3c3af017fcc62998", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.580708, + "spanId": "3c3af017fcc62998", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014625, + "spanId": "32bc17825cb90cbe", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.035875, + "spanId": "a7a080f77ac6dbc3", + "parentSpanId": "84e5273d7328d97e", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.557125, + "spanId": "84e5273d7328d97e", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013042, + "spanId": "1ab8b3702b9cae58", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.294125, + "spanId": "3f84c407d156523b", + "parentSpanId": "e70077955191232f", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.566584, + "spanId": "e70077955191232f", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017959, + "spanId": "8cbe04050537068b", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.515125, + "spanId": "df2777fa0857f7ed", + "parentSpanId": "c58f8979a201ce46", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.866791, + "spanId": "c58f8979a201ce46", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016583, + "spanId": "2861c402508b345f", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.939834, + "spanId": "815c6c0fcaa9050b", + "parentSpanId": "c76aff3d30d3dd6b", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.424084, + "spanId": "c76aff3d30d3dd6b", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015833, + "spanId": "9d072a57fac39dcf", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.1085, + "spanId": "1eafb0178f2e69dc", + "parentSpanId": "6a70ca177da1a41c", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.468583, + "spanId": "6a70ca177da1a41c", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.consume", + "durationMs": 257.566333, + "spanId": "6c0bc7cd5dd496fd", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + } + ] + }, + { + "label": "warm-20", + "elapsedMs": 595.2773749999906, + "nativeAssetMs": 244.01191699999617, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1171095552, + "cpuUserMs": 315.163, + "cpuSystemMs": 504.002, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "02efd58077f4cfcd7b4843d365ae2115", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 89.567, + "spanId": "4f637be809c56234", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.rows", + "durationMs": 23.273334, + "spanId": "f356f04c031a3d04", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023958, + "spanId": "9a20808ed5bd9b83", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.060166, + "spanId": "bab8c6d90715408e", + "parentSpanId": "f5740b94daf6f691", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.699541, + "spanId": "f5740b94daf6f691", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017708, + "spanId": "36f545ca1e08c604", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.15025, + "spanId": "5805c18d2b0505d6", + "parentSpanId": "f329845811fc2f89", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.08375, + "spanId": "f329845811fc2f89", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031333, + "spanId": "86d67668e78f56e5", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.966, + "spanId": "af57aa15ab8a3c20", + "parentSpanId": "6a0b37e31264fa20", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.308833, + "spanId": "6a0b37e31264fa20", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013459, + "spanId": "08b13b5562c6f20b", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.451083, + "spanId": "dac14f954701a7b8", + "parentSpanId": "554024b6fd76baa6", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.046625, + "spanId": "554024b6fd76baa6", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016416, + "spanId": "b6e37f6a184c39d4", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.281291, + "spanId": "a27104b817cdfa8b", + "parentSpanId": "4172e7bfbfbda9cd", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.304041, + "spanId": "4172e7bfbfbda9cd", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014667, + "spanId": "a5e1d49f7016cc43", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.582833, + "spanId": "070eac96c83d0ed6", + "parentSpanId": "a76a7645e5f1e1a5", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.747083, + "spanId": "a76a7645e5f1e1a5", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018334, + "spanId": "462adb15ba3c7e7e", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.888458, + "spanId": "0fbb810cd1f1b35e", + "parentSpanId": "182253a7aec6e461", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.914542, + "spanId": "182253a7aec6e461", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01775, + "spanId": "2c5ea754704ebe14", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.532084, + "spanId": "2969bbddb5d7e004", + "parentSpanId": "3a0aefd37610df57", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.345333, + "spanId": "3a0aefd37610df57", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013042, + "spanId": "49ab817d6a1f7bac", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.377333, + "spanId": "e0f55d8dac1d32f1", + "parentSpanId": "01d07244ac23cbce", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.841209, + "spanId": "01d07244ac23cbce", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020959, + "spanId": "ff24ef87601177da", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.805709, + "spanId": "43451b7c272c2931", + "parentSpanId": "0a61252d03a8c331", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.472042, + "spanId": "0a61252d03a8c331", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.consume", + "durationMs": 268.224875, + "spanId": "1e8db9674aaf49d6", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + } + ] + }, + { + "label": "warm-new-service", + "elapsedMs": 645.1124160000036, + "nativeAssetMs": 276.84833400000934, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1171111936, + "cpuUserMs": 354.318, + "cpuSystemMs": 557.383, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "196df4f3a913e668a19bec5b4e8712cc", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.46075, + "spanId": "8f0419b5e76488c0", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.rows", + "durationMs": 23.6435, + "spanId": "b6839a7d154544e3", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.030958, + "spanId": "ab99e1a18480bde3", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.16, + "spanId": "cbf9578fb6d65742", + "parentSpanId": "d7b003ec7bcba20c", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.383917, + "spanId": "d7b003ec7bcba20c", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014625, + "spanId": "372aa4b8582a91d0", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.738417, + "spanId": "edff35e5e9dcae7a", + "parentSpanId": "c6946634c6de78b0", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.657417, + "spanId": "c6946634c6de78b0", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.049083, + "spanId": "e2c1ea11ccdbb3e7", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.05575, + "spanId": "e712191554558e6c", + "parentSpanId": "621beef0cc725691", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.012375, + "spanId": "621beef0cc725691", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020833, + "spanId": "ec045afdf6a938c2", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.75625, + "spanId": "3cdcc7dc7e8fd28d", + "parentSpanId": "d4ff2e88fff43840", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.678375, + "spanId": "d4ff2e88fff43840", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014458, + "spanId": "a92e593f56c85b6a", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.051625, + "spanId": "45aa01309fd53b1d", + "parentSpanId": "473848b8348e97a5", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.856584, + "spanId": "473848b8348e97a5", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017125, + "spanId": "43d6e82c6465e57f", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.992041, + "spanId": "2cde5272088c3d65", + "parentSpanId": "81d5bacbd7518e93", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.005625, + "spanId": "81d5bacbd7518e93", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.060834, + "spanId": "6364e4cf739829d0", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.14825, + "spanId": "9a9582b06d2c601b", + "parentSpanId": "69dfc28a1b8dca0a", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.869459, + "spanId": "69dfc28a1b8dca0a", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015042, + "spanId": "e87803d36ea735fd", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.808791, + "spanId": "968522a5048d3ec3", + "parentSpanId": "fe4fad0f31789cb3", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.705458, + "spanId": "fe4fad0f31789cb3", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014875, + "spanId": "75d19610b872698a", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.450458, + "spanId": "e0fc9cb907209b59", + "parentSpanId": "32e719e16f571bca", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.3125, + "spanId": "32e719e16f571bca", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017208, + "spanId": "8f04af3139c4eaa4", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.80975, + "spanId": "838d39b4b1043455", + "parentSpanId": "01a7229a9f823451", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.789333, + "spanId": "01a7229a9f823451", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.consume", + "durationMs": 300.645458, + "spanId": "468d74c9b84e8a42", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + } + ] + } + ] + } + ] + } +} diff --git a/doc/plans/2026-10-28-skill-package-storage.md b/doc/plans/2026-10-28-skill-package-storage.md new file mode 100644 index 00000000000..c3ec1d26fc0 --- /dev/null +++ b/doc/plans/2026-10-28-skill-package-storage.md @@ -0,0 +1,444 @@ +# Move installed skill packages out of PostgreSQL + +Status: implemented in this checkout, including automatic conversion, exact-commit +recovery and cache retention. Focused verification passed; the broad test run +did not complete cleanly (details below). No production changes. +Operational contract: [skill package storage](../skill-package-storage.md). +Measured results: [benchmark receipts](2026-10-28-skill-package-performance-results.json). +Investigation date: 2026-10-28. Code inspected: `1ed0af41867ba40fd1d23292d395588d18c2fcbe`. + +## Decision + +Store the exact audited installed package in the existing storage backend: S3-compatible +object storage for hosted instances, persistent local storage for local installations. +PostgreSQL retains package identities, storage receipts, versions, source tracking, +permissions and run pins. A bounded, verified disk cache supplies execution and file reads. + +New Git packages retain an immutable recipe with repository identity, exact commit, +included-file manifest and packager version. Authored and unproven legacy packages +retain independent durable bytes. Also retain the installed package's content hash: the package can exclude unsafe, oversized or unsupported files and can be +a reduced repository subset. A commit SHA alone does not identify that transformation, +provide the bytes when upstream disappears, or represent an edited local copy. + +Preserve today's packaging and access behavior during the storage migration. In particular, +do not switch all imports back to directory-only mode or infer a complete dependency +graph from Markdown references. The original dependency-loss bug was real. + +The first delivery removes file-content traffic from PostgreSQL for migrated repository +packages, avoids repeated remote reads of the same immutable package, and stops creating +packages for repositories with no installable selected skills. Deduplicating the final +per-skill runtime transport is a separate, explicitly scoped follow-up. + +## What the trace established + +### Import and refresh + +1. Project workspace creation/update calls `enqueueProjectSkillSource` inside the + workspace transaction. An eligible GitHub URL is sufficient to enqueue; this happens + before discovery. Existing legacy projects have an explicit adoption path. +2. `skillSourceSyncService.importBinding` checks authority, reserves bounded work credits, + and requests `packageMode: 'repository'`. The scanner resolves a commit and uses a + shallow, temporary Git repository to list paths and read blobs. +3. `github-skill-source.ts` discovers entrypoints, builds and audits package files, and + filters unsupported/unsafe files. Repository mode normally includes the supported + tree, with a skill-folder-plus-reference fallback for oversized repositories and a + further skill-only fallback. It does not store Git history or `.git` metadata. +4. `skillSourceService.publish` hashes `scan.repositoryFiles` and inserts their **contents** + into `company_skill_repository_snapshots.file_inventory`. Deduplication is by company + and content hash. The insert happens before the selected-valid-skill loop; a nonempty + package can therefore be stored with no discovered or selected usable skills. + `ON CONFLICT DO NOTHING` avoids a duplicate row, but still sends the proposed file + array to PostgreSQL whenever publication reaches that insert. The earlier unchanged + commit/tree shortcut avoids this path for qualifying automatic checks. +5. Each selected skill version references that snapshot and its original entrypoint. + Publication also passes `candidate.files` into `createVersion`, which persists those + contents again in `company_skill_versions.file_inventory`. Thus the full repository is + shared, but the selected skills' own files have an additional persisted copy. +6. Automatic refresh already skips downloading when the commit or root tree is unchanged. + It uses saved setup authority, leases, budgets and five-minute deadlines. An included + file change changes the package hash and versions every selected skill. This conservative + invalidation preserves access to shared and dynamically chosen dependencies. + +The similarly named fields are different: `company_skills.file_inventory` and +`company_skill_sources.repository_package.files` are file metadata; repository snapshot +and version `file_inventory` fields contain actual contents. Moving the former alone +would not address the measured payload. + +Source pointers: [project attachment](../../server/src/services/projects.ts), +[binding and identity](../../server/src/services/project-skill-source-bindings.ts), +[worker](../../server/src/services/skill-source-sync.ts), +[scanner](../../server/src/services/github-skill-source.ts), +[publication](../../server/src/services/skill-sources.ts), +[snapshot/version schema](../../packages/db/src/schema/company_skills.ts). + +### Reads and execution + +`company-skills.ts::runtimeVersionInventory` selects the entire repository snapshot, +then `repositorySkillProjection` builds a generated discovery wrapper plus the full +original tree under `.paperclip-repository/`. + +| Consumer | Current unnecessary or amplified work | Replacement | +| --- | --- | --- | +| Runtime preparation | Loads JSONB before checking the materialized directory and before entering the per-version shared promise | Resolve a small package descriptor; coalesce package loading before I/O; validate disk files against hashes | +| Skill audit | Materialization loads the repository; audit requests the inventory again | Reuse the verified package handle and manifest; audit actual delivered bytes | +| Skill detail/file tree | Loads contents, then discards them to keep paths and kinds | Read the package manifest only | +| One supporting file | Falls back to loading the whole repository inventory | Read the requested file from the verified local package cache | +| Fork, create-from-existing, disconnect-and-keep | Projects repository files; editable versions can persist a full projected tree inline | Copy from a package handle; save resulting snapshots to the same external package store | +| Cache retention | `hasRestorableSnapshot` reads repository JSONB on a process-cache miss to validate entrypoints | Verify external durability before eviction; never evict the last usable copy on storage failure | +| Version API / Studio test context | Version objects and one heartbeat join directly consume inline version inventories | Metadata-only internal queries plus explicit package hydration where content is required | + +The existing per-version disk cache does **not** prevent the initial database read. +Existing cache locks serialize publication, pinning and eviction, but do not turn +repository identity into a shared content cache. + +Each selected entrypoint also gets its own full runtime copy. Native remote launch loops +over skill bundles and stages each directory separately. Moving storage alone does not +remove this disk/transport multiplication. Never assume 79 installed skills means all +79 are assigned or transferred on every run. + +Source pointers: [runtime, audit, browsing and version creation](../../server/src/services/company-skills.ts), +[projection](../../server/src/services/skill-repository-package.ts), +[cache locking and restorability](../../server/src/services/skill-version-cache.ts), +[Studio context](../../server/src/services/heartbeat/run-preparation.ts), +[remote staging](../../server/src/services/native-runtime/remote-launch-assets.ts), +[company export](../../server/src/services/company-portability.ts). + +### Evidence and limits + +The preceding Railway investigation recorded six snapshot reads returning **93.6 MB** +in a 15-minute window: **12.94%** of observed PostgreSQL response bytes. A later metadata +inspection found a package containing **1,349 files**, **22,611,267 serialized JSON bytes**, +and **79 current referencing skills**. These are prior observations, not a fresh benchmark +of this proposal. The measured build was `71b448b9`; the later example was observed on +`b25f3e2a`. The principal import/runtime files inspected here match the earlier linked +`2e5bbde8` revision. Do not extrapolate either capture to every current instance. + +Serialized JSON bytes are neither compressed on-disk PostgreSQL size nor raw repository +size. PostgreSQL can compress/store large values out of line using +[TOAST](https://www.postgresql.org/docs/current/storage-toast.html). Selecting the complete +inventory still returns that value to the application. New snapshot payloads also add +database storage, WAL, backup and decoding work; those costs were not quantified here. + +Removing this traffic does not establish a 12.94% latency improvement, solve all Railway +slowness, or justify a specific CPU reduction. The capture measures bytes, not causality +for every slow request. + +## Why this implementation arose + +- [#14713](https://github.com/paperclipai/paperclip/pull/14713), September 30, introduced + GitHub-synchronized skills on the existing immutable-version foundation. +- [#15731](https://github.com/paperclipai/paperclip/pull/15731), October 12, fixed missing + shared scripts, policies, sibling skills and credits by preserving repository packages. + It explicitly chose shared database snapshots and self-contained per-skill delivery. +- [#16066](https://github.com/paperclipai/paperclip/pull/16066), October 13, extended that + behavior to automatic project imports and refresh. +- [#16308](https://github.com/paperclipai/paperclip/pull/16308) preserved referenced files + for oversized repositories. [#16433](https://github.com/paperclipai/paperclip/pull/16433) + subsequently made old disk copies disposable, with database restoration as the guarantee. + +The #15731 description reports strong byte/mode, offline, pinning and real runtime proof, +and explicitly acknowledges per-skill runtime duplication. Its inline reviews caught +audit/projection mismatches, lost binary/mode information, unsafe files, expensive regexes, +and per-blob Git process overhead. The review was not indifferent to correctness or cost. + +The inspected acceptance evidence does not establish a bound on PostgreSQL response bytes +for warm preparation, one-file browsing, or multiple entrypoints sharing a large package. +The supported explanation is that a legitimate dependency-preservation feature reused a +content-bearing version contract, while performance proof focused on acquisition and +functional delivery. Automatic adoption enlarged the affected workload. Original author +sessions were not identified, so this is not a claim about unrecorded reasoning. + +The preventive change is a storage boundary whose normal return type cannot contain a +repository's bytes, backed by real SQL-byte and storage-read assertions for shared packages. + +## Proposed implementation + +### 1. One package service, existing storage providers + +Add an owning service such as `skill-package-store.ts` for encoding, publication receipts, +manifest reads, verified extraction and cache acquisition. `company-skills.ts` consumes it; +the heartbeat only asks for a prepared skill. Provider details stay under `server/src/storage`. + +The existing [storage API](../../server/src/storage/types.ts) supports streamed uploads, +reads, HEAD and deletion, with [local](../../server/src/storage/local-disk-provider.ts) +and [S3](../../server/src/storage/s3-provider.ts) providers. Reuse this infrastructure. +Do not put packages in user-visible issue attachments or give them attachment retention. + +Use a versioned `tar+gzip` bundle of raw file bytes and a separate bounded JSON manifest. +The manifest contains original path, kind, byte length, SHA-256 and executable flag for +each file. Archive entries live under `files/`; reserved format metadata stays outside +that namespace. Preserve binary bytes and supported modes. No base64 repository JSON is +needed in the stored archive. Stream encoding/compression to a bounded temporary file, +hash it, then stream upload. Do not synchronously compress on the server event loop. + +Keep the existing semantic content hash for migrated packages, so migration does not +create new skill revisions. Separately record archive and manifest SHA-256 values and +format version. Do not silently change the current hashing order or rely on an S3 ETag +as a content checksum. Any future hash-algorithm change needs its own versioned identity. + +A manifest is sufficient for a file tree; a cold file read may fetch the archive once. +An archive deliberately favors simple bulk materialization over random remote file reads. +If cold single-file traffic proves material, add an indexed/range-readable format later. +Do not start with thousands of remote objects and database rows per package file. + +### 2. Metadata schema and contracts + +Extend the existing snapshot row with storage state/format and a receipt containing the +provider, server-owned archive/manifest keys, hashes, compressed/uncompressed sizes and +file count. Keep the existing snapshot ID and `(company_id, content_hash)` uniqueness. +Add company-scoped relational constraints where a new package reference is introduced. + +During migration a snapshot is one of: legacy inline; inline plus verified external copy; +external only. An explicit discriminator distinguishes absent payload from an empty +package. Keep pending upload attempts separate from externally readable versions, using +small preparation receipts/leases in the owning snapshot storage metadata. A final version +may reference only a ready package. + +Repository-backed version rows retain their snapshot and entrypoint references but stop +duplicating `candidate.files`. Preserve the directory-local view as a manifest projection. +Existing historical version inventories must be migrated from their actual stored bytes; +do not assume an older or checkpointed inventory exactly matches a current projection. +If it differs, store that inventory as its own immutable package. + +Support ordinary local, directory-only and forked versions with the same package reference +in a second compatible writer path. Keep a bounded inline legacy reader until backfill +finishes. Small searchable skill Markdown/metadata can remain on `company_skills`; the +goal is to remove file-tree payloads, not prohibit useful database text. + +Split internal `VersionMetadata` from content-bearing responses. Public +`CompanySkillVersion.fileInventory`, the version routes, Studio and export consumers +currently expect contents: preserve their response behavior through explicit hydration +from storage, or add an opt-in metadata view and move clients to it. Never return an empty +inventory to an old caller merely because the database payload was cleared. A storage +location or opaque digest is not an authorization capability and need not be exposed. + +### 3. Publication without long database transactions + +```text +authorize and scan immutable Git revision + -> audit/filter the installed package + -> obtain a bounded preparation claim for (company, content hash) + -> write manifest + archive to unique attempt-owned storage keys + -> verify stored lengths, hashes and manifest/file agreement + -> short DB transaction: recheck source revision/lease/authority; + publish ready receipt, versions, current pointers and activity together + -> settle preparation receipt and release temporary files +``` + +Perform provider I/O outside the publication transaction. Refresh's current source and +lease fencing still decides whether the result may be published. A failed upload keeps +all installed versions. A failed or uncertain commit must be reconciled before cleanup; +never delete an object simply because the transaction call returned an error. + +Use immutable unique attempt keys, with logical deduplication in PostgreSQL. This matters: +`createStorageService.putFile` currently deletes its upload key on failure. Reusing a +single deterministic key across writers could let a losing/failed writer delete a winner. +Do not use that API as a create-if-absent content-addressed store. A concurrent winner's +receipt is retained; the losing attempt's own keys become cleanup candidates only after +its outcome is known. Preserve pending receipts through restart and company deletion. + +Amazon S3's [strong read-after-write consistency](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) +helps verify a successful object write, but does not make an S3 write and SQL commit atomic. +Qualify other supported S3-compatible backends separately. Pass cancellation through the +storage facade; its current public read API omits the provider's AbortSignal capability, +and uploads need bounded cancellation/settlement too. Avoid abandoned streams after timeouts. + +### 4. Shared verified cache before per-skill projection + +Use a cache identity containing instance/storage namespace, company, format and package +content identity. Reauthorize the skill/version before every use. Deduplicating content +within a company must not expand a private project's audience; authorization stays on +the referencing skill/source, including forks and detached copies. + +Acquire a package-scoped shared promise **before** opening legacy payloads or object +storage. One cold request fetches/verifies/extracts the package; concurrent consumers reuse +it. Different service instances in one process share this work. Use filesystem locking +and atomic publication for processes sharing a volume; replicas with separate disks may +each need one fetch. Keep cache ownership distinct from authorization and version pins. + +Validate compressed size, expanded size, file count, normalized paths, duplicate/colliding +entries, per-file digest and executable mode. Reject links, devices and traversal; never +run repository code on extraction. An extracted directory or `.complete` marker alone +is not proof of byte integrity. Warm runtime validation compares actual disk files with +the hashed manifest, including unexpected files and modes, without rereading SQL payloads. +Audit continues to inspect actual content and may require new policy scans independently +of package immutability. + +Keep the current self-contained `.paperclip-repository` skill layout initially. Build it +from the shared package cache and reuse the manifest in materialization and audit. Preserve +the existing company cache lock through final publication and durable run pinning. Stage +slow downloads before that lock, then recheck authority/version identity inside it; if +the caller already holds an outer preparation lock, change that owner deliberately rather +than weakening pin safety. Cache eviction must respect active handles during staging. + +Bound cached disk bytes, active downloads, expanded temporary bytes and in-memory manifests; +do not introduce an unbounded process-wide map of decoded repository arrays. The existing +61 MiB package cap is not a bound on concurrent copies or historical cache growth. + +### 5. Avoid content work when nothing can be installed + +After listing the immutable tree, return a successful empty discovery when no valid +entrypoint path exists, before batching support blobs and constructing a repository package. +Persist the source's checked commit/tree and next check, not its contents. The initial +shallow Git fetch still transfers Git objects; do not claim zero acquisition cost. + +At publication, require at least one selected, valid candidate needing a package. A scan +with only excluded/invalid skills can retain candidate metadata and warnings without a +durable payload. A scan in which upstream skills disappear retains previously installed +versions and their packages. A later commit can still discover and import a new skill. + +Keep current unchanged-commit/tree checks and work-budget accounting. Changing the +five-minute refresh policy or adding provider events is independent work, not a prerequisite +for moving bytes out of PostgreSQL. + +## Alternatives considered + +| Alternative | Assessment | +| --- | --- | +| Store SHA; clone from upstream on use | Small database, but loses offline/deleted/revoked-upstream recovery and adds execution-time network work; does not reproduce filtered packages by itself | +| Durable bare Git mirror with retained refs | Useful for acquisition deduplication; requires durable backup, pruning/GC rules, isolation and a filtered-export contract; does not naturally cover edited/generated skills | +| Self-contained Git bundles in object storage | Viable for raw Git retention; Git distinguishes self-contained bundles from incremental bundles with prerequisites. More Git state than the installed-package contract needs | +| Compress JSONB or add a large RAM cache | Can reduce a symptom; keeps database payload storage and cold-read cost, and adds heap pressure | +| Always keep only a skill directory | Reintroduces known missing shared-file dependencies | +| Per-file content-addressed storage | Better cross-version dedup and single-file access; adds object count, manifest graph and garbage-collection complexity before that benefit is measured | +| Immutable package archive + manifest + disk cache | Recommended first implementation: existing providers, bounded bulk I/O, exact installed bytes, stable version references | + +The [Git bundle documentation](https://git-scm.com/docs/git-bundle) describes prerequisite +verification. A bundle or mirror is a real retained copy; a remote SHA is only a reference. +A reusable Git acquisition cache remains worthwhile if profiling shows repeated shallow +fetches dominate, but it must preserve the scanner's credential and caller isolation. + +## Automatic conversion, acquisition and retention + +The accepted follow-up removes the manual migration requirement. New writes default +to external packages, and a finite delivery worker converts legacy rows, independently +verifies storage and then clears duplicate payloads. It uses indexed row deadlines, +transaction notifications and restart reconstruction, rather than per-turn work or +an empty-table polling interval. The CLI is optional diagnostics. See the current +[operational contract](../skill-package-storage.md) for upgrade/rollback details. + +New Git imports persist a restore recipe alongside the package: repository identity, +source authorization reference, exact commit, precise included-file manifest and +packager version. Warm queries exclude this larger metadata. A cold miss first uses +local verified files, then stored bytes, then legacy DB bytes, then the original +Git commit using current source authorization. Every recovered file must match the +saved size/mode/hash; moving branches and current filtering rules cannot change it. +Historical packages without a provable recipe remain durable instead of inferring +an origin from mutable current skill metadata. + +Unused Git archives become eligible after 30 days, keeping current versions, explicit +pins and active/recovering runs. Metadata and recipes survive expiry. Retired storage +keys have another 30-day deletion grace. Locally authored and edited packages remain +durable while referenced. Canonical extraction retains its bounded cold-fill eviction; +version directories use existing retention; native runtime assets now have a 30-day +idle lifetime, pin-aware cleanup and a shared publication/deletion lock. A pin can +conservatively defer cleanup until the next relevant mutation/use or restart. + +Source and package writes remain company-scoped. Saved grants are rechecked on Git +acquisition, with a persisted five-minute failure backoff and existing singleflight. +No upstream network request, storage HEAD, cleanup scan, or TTL write is added to +warm package preparation. Native access timestamps are coalesced hourly on disk. + +Lifecycle ownership remains in the package/native asset services. `index.ts` only +registers them with the existing coordinator. `migration_at`/`maintenance_at` and the +existing deletion receipts are durable truth; no additional worker queue table was +introduced. Conversion retains the original bytes and may wait for ingress without +blocking idle sleep. Optional retention may wait for the next ingress wake. This does not establish deployed sleep or a Cloud alarm contract. + +## Delivery slices and proof + +| Slice | Owning files/area | Exit condition | +| --- | --- | --- | +| A. Storage boundary and compatibility | New `skill-package-store.ts`; storage facade; DB/shared version descriptors; `company-skills.ts` readers | Real local and S3-compatible upload/restore round trips; old rows work; metadata reads contain no file bytes; no behavior change yet | +| B. Publisher and runtime | `skill-sources.ts`, scanner, version/fork/copy writers, package cache, cache retention | New package bytes never enter SQL; two or more selected skills share one cold package load; warm preparation does not reload it; zero-skill scan persists no package | +| C. Consumers and migration | Version routes/client contracts, Studio context, export/disconnect, bounded migration command | Old API content behavior preserved; historical pins and copies restore; full current/legacy consumer matrix passes; backfill can restart safely | +| D. Qualification and rollout | Existing skill suites and runner journey; operational docs/maps | Byte/performance evidence, fault injection, staged backfill, rollback boundary and cold recovery demonstrated | + +Slices A–D form the complete database-storage change. Keep runtime transport deduplication +separate: add a shared package bundle plus entrypoint descriptors to the runtime context, +transfer each package once, and construct each discovery entrypoint in the destination. +Preserve the existing fallback for adapters that require independent directories. Do not +use host-only symlinks or shared writable hardlinks as a shortcut; qualify read-only mounts, +copy-on-write copies or destination-local materialization against each supported transport. +This later change reduces transfer/disk multiplication; it is not needed to remove DB bytes. + +Use the existing `skill-sources`, `skill-version-cache`, `skill-version-retention`, project +skills and native runner suites. Add a representative large synthetic package and an +assembled PostgreSQL/storage benchmark, not mocks that eliminate serialization and I/O. + +| Scenario | Required evidence | +| --- | --- | +| 1,349 files / approximately 22.6 MB JSON / 1, 2 and many entrypoints | Record exact fixture size, actual PostgreSQL DataRow bytes, object GET count/bytes, CPU, peak RSS, disk bytes and preparation duration; sizes need not reproduce private content | +| One cold preparation using several skills from one package | Zero repository file-content bytes in SQL; at most one payload download per package per shared cache scope; equal delivered bytes/modes | +| Warm preparation, audit and restart with intact disk cache | Zero repository payload bytes from SQL; zero payload GETs while cache is valid; actual integrity checks retained | +| File tree and one-file read | Tree reads manifest only; first file read may fetch one archive, repeated reads reuse it; unrelated package contents never pass through SQL | +| Unchanged refresh and no-skill repository | No new package/version payload; empty scan remains discoverable on a later commit; existing installed content survives removal/failure | +| Storage failure, corrupt archive/manifest/cache, cancellation | No partially published version; old installation remains; no silent SHA refetch; unrelated requests remain responsive | +| Concurrent import, uncertain commit, process crash, deletion | No winner object deleted; persisted attempt reconciliation; stable IDs and source lease/revision fencing | +| Private project, revoked access, detached/forked content | Fresh application authorization on cache hits; no company/project boundary widening | +| Pins, cold restore, Studio, forks, disconnect, export, retention | Equivalent supported files, binaries/modes and discovery policy; historical semantics preserved without upstream | +| Backfill interrupted at each boundary | Retry changes no version IDs or contents; no early payload clearing; dual-reader rollback works | + +Report SQL byte reduction separately from object-store traffic and execution latency. +The target is **zero package file-content bytes in PostgreSQL responses for migrated +packages**, not zero database queries or zero network bytes. No percentage CPU/latency +saving is claimed until measured on the replacement. + +Update `doc/PRODUCT.md`, `doc/SPEC-implementation.md`, `doc/SPEC.md`, `doc/DATABASE.md`, +`doc/DEVELOPING.md` and recoverable-cleanup documentation to replace the database-byte +restoration guarantee with durable-package restoration. Update affected skills/version +feature and access map entries, and run `node scripts/check-maps.mjs`. Generate schema +migrations with `pnpm db:generate` and follow migration snapshot rules. Run focused checks +first; before a PR-ready implementation handoff run `pnpm -r typecheck`, `pnpm test:run` +and `pnpm build`, plus the affected real runner journey. See the operational contract +and verification record below for the checks actually completed in this checkout. + + +## Implementation verification record + +Implemented the storage migration, dual reader and guarded writer, bounded migration +CLI, deletion receipts, package/runtime caches, metadata batching and native asset +reuse. The approved follow-up adds default-on writes, automatic verified conversion, +immutable Git restore recipes, persisted acquisition backoff, automatic object cleanup +and native asset retention. The public API continues to hydrate explicit file-content responses. New source +packages are prepared before publication transactions; existing generic authoring +transactions keep their serialization boundary and use an independent upload connection. + +- Full workspace typecheck passed; the server was rebuilt after the final refinements. +- Full repository build passed. Migration numbering/safety and map contract checks passed. +- Focused package, source, company-skill, retention/cache, concurrency, heartbeat preparation + and native context suites passed. The final migration/retention rerun passed 36 tests; + the final project-source/authoring-route rerun passed 92 tests. +- Real loopback S3-compatible multipart upload and historical restoration passed; its + disposable server/data were removed after verification. +- The benchmark records 60 warm samples across 1, 2 and 10 selected entrypoints, plus + cold and fresh-service cases. It includes actual PostgreSQL wire bytes and native + asset creation. It does not prove deployed Railway latency or a live agent turn. +- The serial full-suite invocation was deliberately cancelled and replaced with the + repository's supported CI groups/shards for the same inventory. The broad run + recorded setup timeouts and assertions in the skill concurrency, native recovery, + workspace and other suites. It was stopped to isolate those failures before the + remaining chat, route and workspace groups completed. The skill concurrency case, + two representative recovery cases and the workspace assertion all passed on isolated + reruns. This is **not** a clean full-suite result or complete CI qualification. + +Per-entrypoint runtime/transport deduplication remains the separately scoped follow-up +from this design. Writers and automatic conversion now default on; the explicit false +override remains available for a coordinated reader-first rolling upgrade. + +### Automatic lifecycle follow-up + +The user approved automatic conversion, acquisition on miss and cleanup. This is +implemented without a required data-migration command. Final proof: 30 focused tests +passed, followed by 15 package/worker cases including additional missing-object and +retention regressions; 73 source/native/retention integration cases passed. Idle-safety +coverage passed, workspace typecheck/build passed, and the final server rebuild passed. +The supported fresh-cluster test path bypassed shared-template startup failures. +Two actual regressions found during development—JSONB key reordering during recipe +verification and empty native-cache startup—were fixed and covered in the passing run. + +Forty new warm benchmark samples (one and ten selected skills) plus cold/fresh-service +samples retain five ordinary SQL queries and zero object/body reads. Receipts and timing +limits are in the [operational guide](../skill-package-storage.md#automatic-lifecycle-follow-up-verification) +and the linked JSON. No production cleanup, deployment or data migration was performed. +The earlier broad full-suite limitation remains open. diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index 93f9ccb6fef..d268de28f16 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -2,9 +2,13 @@ The server materializes immutable skill versions under `/skills//__versions__//`. These are disposable -disk copies: the company-scoped version and repository snapshot rows retain the -contents, including binary files and executable flags. Normal skill resolution -rebuilds a missing copy from the database without contacting its upstream source. +disk copies: company-scoped versions reference durable packages in the configured +storage backend, or retain legacy database inventories. Normal skill resolution +rebuilds missing copies, preserving binary bytes and executable flags. New Git packages +can also restore the exact original commit from their saved recipe using current +repository access. Authored/legacy packages without that recipe require independent +external durability before eviction; unavailable storage leaves those caches intact. Shared extracted packages have a +separate cold-fill capacity policy. See [package storage](skill-package-storage.md). ## Skill version caches diff --git a/doc/skill-package-storage.md b/doc/skill-package-storage.md new file mode 100644 index 00000000000..91fdd7c9005 --- /dev/null +++ b/doc/skill-package-storage.md @@ -0,0 +1,307 @@ +# Installed skill package storage + +Installed repository and editable-skill contents can live in the configured storage +backend instead of PostgreSQL JSONB. The database keeps company-scoped package +identities, storage receipts, source tracking, versions, permissions and run pins. +The original repository tree, wrappers, binary bytes and executable flags retain +their existing behavior. New Git packages also retain an immutable restore recipe; +locally authored/edited packages and unproven legacy packages keep a durable copy. + +## Automatic upgrade + +No data-migration command is required. The normal schema upgrade applies migration +0415 (`0415_zippy_zemo.sql`). External writes and automatic conversion are enabled by default. Startup +restores pending work from indexed `migration_at` / `maintenance_at` columns. The +shared delivery coordinator converts four snapshots and four versions per batch, +reading one legacy body at a time. It verifies the hash, uploads, reads the objects +back, validates extraction, then clears the legacy JSONB. Version IDs, revision numbers +and run pins stay unchanged. Failures retain the original bytes and retry after an hour. +Bundled release inventories keep their existing small inline validation contract. + +Readers can use legacy contents while conversion is pending, including when conversion +commits between the metadata read and the content read. Conversion runs separately +from turn preparation. Completed migration history has no recurring scan. Normal +vacuum and backup retention determine when PostgreSQL reclaims physical space. + +`PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED=false` is an explicit compatibility override +for a coordinated rolling upgrade: it disables new external writes and automatic +package maintenance. All processes must support the new reader before allowing +external-only writes or clearing old payloads. Once automatic conversion runs, this +reader is the rollback floor; an older binary needs rehydrated database contents. +Use durable, backed-up local storage or S3-compatible storage for authored packages. +The extracted cache is not that storage backend. + +Deleting an editable file or folder keeps a temporary copy outside the skill until +the transaction commits. Upload or database failures restore the original subtree, +including binary files, permissions, empty folders and files omitted from inventory. +Successful deletions remove the temporary copy; historical versions retain their +existing package bytes. If the commit reply is lost, recovery checks the saved +version under the mutation lock before restoring. A committed deletion stays +deleted. If the database is unavailable or the skill has since changed, recovery +preserves the backup and logs its location instead of overwriting newer work. + +The optional `server/scripts/migrate-skill-packages.ts` command remains available for +diagnostics and controlled batches. It defaults to dry run, supports +`--phase=snapshots|versions|orphans|deletions`, `--company=`, `--limit=1..50`, +`--after=`, `--apply` and `--clear-legacy`. It is not part of normal installation +or upgrade. Never point it at another instance's database/storage configuration. + +## Acquisition and retention + +A missing package follows this order: verified local files, stored archive, legacy DB +contents, then an authorized exact-commit Git fetch. A new Git recipe records repository +identity, the original source, a full commit SHA, packager version, the precise included +file manifest (paths, encodings, modes, sizes and hashes), and the content hash. It is +loaded only on a cold miss, never with warm metadata queries. Recovery reads exactly +those files without rerunning discovery or today's filtering rules. A reused repository +name, wrong commit, changed file/mode/hash, missing original commit, or revoked source +credentials fails explicitly; it never substitutes the latest branch. Concurrent misses +coalesce and failed acquisition has a persisted five-minute backoff. + +After 30 days, unused Git packages with a verified recipe can lose their cached archive +and extracted copy while retaining their identity, manifest and recipe. Current library +versions, explicit agent pins, active runs and unresolved recovery keep their packages. +A source/version change schedules another eligibility check; packages held by a run or +pin are also reconsidered on restart. This is conservative retention, not an immediate +promise to purge on unassignment. Legacy packages without a provable recipe and authored +or edited copies retain their durable bytes as long as a version references them. +A shared package remains durable when any referencing version lacks its own immutable +repository reference. This also conservatively retains per-skill Git imports without +repository projections. Authored preparation permanently marks the storage receipt as +durable, including when it joins a concurrent Git upload. Expiry checks that marker in +its update, so a later Git import cannot weaken an authored version's retention. + +Native runtime assets expire after 30 days without use. An hourly coalesced filesystem +access lease protects preparation through durable run pinning. Cleanup shares the +publication lock, preserves assets referenced by active/recovering runs, and defers if +an unfinished input cannot be interpreted safely. After a pin blocks cleanup, the next +asset use or startup reconsiders it. Missing or damaged copies rebuild from verified +source bytes. The canonical package cache retains its capacity policy below, and runtime +version directories retain their existing pin-aware retention. + +Package/version publication and deletion use transaction-aware work notifications; +local asset use signals its filesystem worker. Both reuse the delivery coordinator, +restore deadlines on startup, pause in drain/standby, and stop on shutdown. With no +pending conversion, retry, or eligible cache deadline they perform no recurring work. +The supported live writer is the owning server; independent SQL writers/other servers +need restart reconstruction or an explicit wake. No permanent DB listener is added. +Conversion retains authoritative legacy bytes until verification succeeds. Conversion +retries and cache/deletion dates are optional maintenance eligibility; they may wait +until the next ingress wake, including with package storage disabled. They do not +block idle sleep or claim an external Cloud alarm or measured host suspension. +Each object-deletion receipt has a 30-second storage deadline. Shutdown cancels +pending delete/existence requests; incomplete cleanup retains the receipt for retry. + +## Execution and cache behavior + +- New packages store one gzip tar archive and one manifest per company/content hash. + The manifest contains paths, kinds, encodings, executable flags, byte sizes and + SHA-256 values. PostgreSQL stores keys, hashes, sizes and ownership metadata. +- Preparation uses metadata-only version/package queries. Multiple selected versions + are batched, and a shared repository descriptor is read once per preparation. +- Known selected repository skills skip unrelated library reconciliation. Selected + rows and project visibility are still read fresh. Public rows are constrained with + `access_source_id IS NULL` at the final read, so a concurrent scope change fails closed. + Aliases, legacy operational skills, missing installations and mutable local sources + retain the discovery/reconciliation path. +- A cold package download is coalesced in process; the existing company cache lock + serializes cross-process restoration and pinning. The archive and each extracted + file are verified before atomic publication. Extraction rejects unexpected entries, + path escapes, links, duplicate paths, wrong sizes, modes and digests. +- Warm checks inspect directory entries and file identity/size/mode/mtime/ctime. + Unchanged files avoid content reads and hashing. Changed identities are hashed again; + altered runtime copies are rebuilt. No marker alone establishes integrity. +- Native asset packaging also caches manifests and asset references, checking both + the selected source and delivered asset on reuse. It retains no repository bodies. + Source edits invalidate the reference; missing or corrupt copies rebuild from verified bytes. +- Concurrent cold package I/O is limited to two operations, with a queue of at most 32. + Cached version inventories and file previews have a separate two-reader, 32-waiter + bound, so slow storage cannot prevent verified local reads. Both queues honor deadlines + and cancellation; an overloaded local-read queue does not trigger a storage download. + Identical cold version/file reads are coalesced. Verified scratch restores repopulate + the shared cache under a short, nonblocking publication lock, so subsequent reads + reuse local bytes. Eviction verification runs outside that publication lock. + Cold administrative responses also have a two-response, 32-waiter bound held + through post-fill cleanup, so decoded inventories cannot accumulate behind storage work. + Caller-scoped metadata is read before taking that permit. Acquisition and cleanup + use a private database connection, so queued transaction readers cannot deadlock + the ordinary pool, including instances configured with one connection. + Legacy repository bodies load only inside the I/O limit, including when deriving + a missing manifest. Waiting readers retain metadata rather than repository contents. + Packages share the Git scanner's limits of 10,000 files and 100 MiB decoded bytes. + Recovery splits reads within the Git reader's 9,990-file and 61 MiB batch limits. + Individual manifests are limited to 48 MiB; the manifest cache holds 16 MiB of serialized data. + Repository projection metadata is shared; generated + wrappers/policies have a 16 MiB budget and 256-entry limit. The stat/digest cache holds + at most 128,000 file records, with 24 concurrent file checks. + Native skill asset manifests hold at most 32,000 file records and 128 entries. +- Shared extracted packages target 512 MiB and 64 directories per company, trimming + on cold fills only, with at most two eviction attempts and a ten-second storage + verification budget per fill. This is a soft limit during a storage outage: cleanup preserves + the last recoverable copy. Version projections retain the existing pin-aware cleanup. + Abandoned package staging directories receive a one-hour crash grace. +- Internal package caches, their files, and containing directories cannot be + imported as local skill sources. Fork the installed skill to retain its original + access restrictions. + +Each entrypoint still has a self-contained runtime tree, and remote staging can +duplicate those trees. This change does not introduce a new runner transport format +or cross-entrypoint asset deduplication. + +## Publication, cleanup, and backups + +Source imports prepare and verify packages before opening their publication +transaction. Generic authoring calls can arrive inside an existing edit/idempotency +transaction; these retain that serialization lock during upload and use an independent +package connection to avoid a one-connection pool deadlock. No upload happens on the +unchanged warm preparation path. + +Uploads use unique attempt keys. An incomplete attempt has a durable claim with a +ten-minute expiry. Retries after expiry use fresh keys; an ambiguous database result +never triggers deletion of a possibly successful upload. A database trigger records +retired keys in `skill_package_deletions`, including company cascades. Promotion of +an upload to a published receipt does not enqueue its objects for deletion. + +The maintenance phases `orphans` and `deletions` have the same dry-run/cursor controls. +`orphans` only removes unreferenced snapshot rows older than one day whose upload +claim is settled. Foreign keys protect referenced versions. `deletions` rechecks that +no live receipt owns the keys, deletes the two objects, confirms absence, then removes +the deletion receipt. Failed attempts have at least one hour of grace; retired +published objects have 30 days. These phases also run automatically from their saved +deadlines. Do not apply a blanket bucket lifecycle rule: authored packages and legacy +packages without exact restore recipes still require durable retention. + +Database backups must be paired with object backups/history. Do not process deletion +receipts while a retained database backup outside the 30-day window still needs those +objects; extend receipt deadlines or retain object versions according to that backup +policy. A storage provider/bucket/prefix change requires copying and verifying the +objects first. The current storage facade does not find objects in an old bucket. + +## Verification + +`skill-package-store.test.ts` exercises real PostgreSQL/local storage, cold/warm +reads, same-size edits, offline historical restoration, missing durable objects, +upload recovery, company-deletion receipts, cancellation and resumable migration. +A reader that obtained its descriptor before the legacy clear pass follows the new +receipt, including when its old manifest is already cached. Existing source, +version-cache, retention, concurrency, authoring and runtime-context suites preserve +their original behavior checks. `runtime-context.test.ts` checks warm asset reuse, +source edits and repair through the normal native context builder. Additional recovery, +worker-lifecycle and asset-retention tests cover exact-commit restoration, automatic +conversion, persisted retry, idle/drain behavior and rebuilding expired native copies. + +The native preparation benchmark accepts `--package-store=external` and +`--native-assets=true`. It reports elapsed time, PostgreSQL wire response bytes, +root-pool query count, object GET count/bytes, process RSS and preparation spans. +Use `--snapshot-file=` for a representative saved package, +`--actor=agent --staging-shape=true` for authorization and unrelated-library load, +and `--network-delay-ms=5` for an emulated database delay. Synthetic local timings +are regression evidence, not a claim about deployed Railway latency. + + +To generate the same-size synthetic fixture without copying a private repository: + +```sh +python3 - <<'PYTHON' +import hashlib, json +files = [{"path": "skills/diagnostic/SKILL.md", "kind": "skill", "content": + "---\nname: diagnostic\ndescription: Native preparation benchmark\n---\nUse the bundled repository.\n"}] +for i in range(1348): + text = hashlib.shake_256(str(i).encode()).hexdigest(8300) + content = "\n".join(text[n:n + 64] for n in range(0, len(text), 64))[:16600] + files.append({"path": f"assets/{i}.txt", "kind": "reference", "content": content}) +with open("/tmp/skill-package-benchmark.json", "w") as output: + json.dump(files, output, separators=(",", ":")) +PYTHON +pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-skill-preparation.ts --snapshot-file=/tmp/skill-package-benchmark.json --package-store=external --native-assets=true --actor=agent --staging-shape=true --network-delay-ms=5 --selected-skills=10 --warm-runs=20 +``` + +The fixture generator reproduces the recorded file counts and JSON size, with new +synthetic content. Use `--selected-skills=1`, `2`, or `10` for comparable cases. + +## Measured results, 2026-10-28 + +[Recorded samples](plans/2026-10-28-skill-package-performance-results.json) use a +synthetic 1,349-file, 22,799,129-byte JSON inventory. The multi-skill cases add one +entrypoint per additional selected skill. Each case includes native asset preparation, +fresh agent authorization, 155 unrelated library skills, and 5 ms of database network +delay in each direction. The table summarizes all 20 numbered warm samples; a fresh +service instance in the same process was measured separately. + +| Selected skills | Warm median | Warm p95 | PostgreSQL response bytes | Ordinary queries | Object GETs | +| --- | ---: | ---: | ---: | ---: | ---: | +| 1 | 245 ms | 326 ms | 19,288 | 5 | 0 | +| 2 | 298 ms | 332 ms | 20,312 | 5 | 0 | +| 10 | 573 ms | 674 ms | 28,536 | 5 | 0 | + +All warm cases transferred zero package file-content bytes through SQL and zero +object bytes. Query counts exclude the dedicated advisory-lock connection; wire bytes +include the measured proxy traffic. Cold preparation fetched one manifest and one +archive, about 13.45 MB total, even with ten selected entrypoints. Cold elapsed times +were 3.7, 6.3 and 11.0 seconds respectively. These local samples ran alongside other +verification work and do not establish production latency or a CPU reduction against +an unchanged baseline. + +Median process CPU per warm preparation was 83/175/820 ms for 1/2/10 skills; filesystem +checks run concurrently, so CPU can exceed wall time. Process peak RSS was about +887/937/1,235 MB, including the application, synthetic fixture and cold allocations; +these are not incremental cache-memory measurements. Allocated instance file bytes +were about 97/152/596 MB. Runtime/native tree duplication remains visible here and is +a separate optimization. Cache budgets also mean sufficiently large selected sets or +many active companies can evict metadata/digests and require new hashing or asset +assembly; the warm timing is not a constant-time guarantee for arbitrary workloads. + +The real S3-compatible proof used a disposable loopback MinIO server built from its +official `RELEASE.2025-04-22T22-12-26Z` source. An 18,008,626-byte archive containing +22 files (18,000,095 decoded bytes) passed multipart upload, independent read-back +verification and cold restoration. `server/scripts/verify-skill-package-s3.ts` repeats +this test against a loopback endpoint and creates/deletes its own bucket and database. +No production storage or Railway instance was changed. + + +Workspace typecheck, build, migration checks and focused integration tests passed. +The attempted full repository test run did not complete cleanly: parallel shards +reported fixture setup timeouts and assertions, then were stopped for isolation. +The affected skill concurrency assertion and sampled recovery/workspace failures passed +on isolated reruns. The remaining broad inventory is not claimed as verified; see the +[implementation record](plans/2026-10-28-skill-package-storage.md#implementation-verification-record). + +## Automatic lifecycle follow-up verification + +The final implementation passed 30 focused package/recovery/worker/native tests, +then 15 package/worker tests including two additional regressions for missing stored +objects with a live receipt and retaining unproven historical bytes. Source-sync, +version-retention and native integrations passed 73 tests. The idle-safety suite also +passed. Full workspace typecheck/build passed; the final server was rebuilt after the +manifest normalization fix. Migrations and map checks passed. + +The shared PostgreSQL test-template path timed out (and exited a standalone benchmark +before measurement). The passing final runs used the supported +`PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` fresh-cluster path. Earlier retirement assertions +caught PostgreSQL JSONB key reordering; manifest serialization now preserves its v1 +wire encoding after a DB round trip. An empty native cache startup case was also fixed. + +The final 20-sample warm runs measured: + +| Selected skills | Median | p95 | SQL response bytes | Ordinary queries | Object reads | +| --- | ---: | ---: | ---: | ---: | ---: | +| 1 | 256 ms | 302 ms | 19,423 | 5 | 0 | +| 10 | 691 ms | 816 ms | 28,969 | 5 | 0 | + +All warm samples still transfer zero package bodies through SQL and zero object bytes. +The larger restore recipe stays out of those queries. Ten-skill cold preparation was +slower on this busy host; the raw receipts retain that result as well as warm samples. +The earlier full repository test attempt remains incomplete; this follow-up does not +claim a clean full-suite result, deployed Railway measurements, or a live agent run. + +### Administrative version reads + +Version lists, current-version summaries, and test-run detail return file names/kinds +without bodies. Diff and restore fetch only the selected version through its existing +single-version endpoint. Archive reads for those operations and retention checks use +isolated scratch directories, so storage latency does not hold the company preparation +lock. Support inspection reads the requested stored version without Git recovery. + +Package preparation renews a 24-hour publication grace. Publication and retirement +lock the same package row and recheck its receipt/current references before commit; +a prepared receipt retired before publication cannot silently become current. diff --git a/maps/agent-access-map/skills.md b/maps/agent-access-map/skills.md index 71776058621..100a06f8921 100644 --- a/maps/agent-access-map/skills.md +++ b/maps/agent-access-map/skills.md @@ -13,6 +13,22 @@ Each capability needs operation-level review before its entire scope can be qualified. No live agent execution is claimed. Follow the [map contract](../README.md) for format, permission checks, evidence and maintenance. +Cold Git package recovery rechecks the original source's current setup identity, +company membership, policy and repository grant. It never borrows another company's +credentials or substitutes a moving branch for the saved commit. A missing/revoked +source fails with a reconnect error and persisted retry backoff. Warm access still +uses the ordinary skill/project permission check. + +External package storage changes persistence, not agent authority. Runtime selection +still checks current company/project access before consulting immutable package caches; +no storage key is exposed as an agent capability. Public selected rows are constrained +to `access_source_id IS NULL` at their final read. Warm package and native asset caches +hold file metadata, never authorization decisions. The +[concurrency regression](../../server/src/__tests__/company-skills-runtime-concurrency.test.ts) +covers separate actors and revoked project access; the +[package regression](../../server/src/__tests__/skill-package-store.test.ts) covers +company isolation and warm-cache behavior. This does not claim live agent qualification. + ## `discovery` Capability ID: `skills/discovery`. @@ -75,6 +91,12 @@ support a narrower shape than full API. ## `versions` +Version lists and current/test-run version summaries contain file names and kinds. +Use `GET /api/companies/{companyId}/skills/{skillId}/versions/{versionId}` for the +selected version's file contents. Authorization remains company/skill scoped; support +inspection reads only stored bytes for the requested version and does not recover Git. + + Capability ID: `skills/versions`. Review: `baseline`. diff --git a/maps/feature-map/skills.md b/maps/feature-map/skills.md index b3db94c6cc9..279b7d95344 100644 --- a/maps/feature-map/skills.md +++ b/maps/feature-map/skills.md @@ -53,6 +53,12 @@ Automated: [Studio](../../ui/src/pages/SkillStudio.test.tsx) covers UI behavior; Manual: Create or fork a disposable skill, edit instructions, save, and reload. Run a bounded test, respond to any question, and inspect the output. Verify the saved version matches what was tested and failed saves retain the draft. +Version history lists file metadata without downloading archives. Opening a diff +fetches only its selected versions; restoring fetches the chosen version. The +[library UI regression](../../ui/src/pages/CompanySkills.test.tsx) checks deferred +loads; the [package regression](../../server/src/__tests__/skill-package-store.test.ts) +checks that slow archive verification cannot hold up a warm agent preparation. + ### `agent-enablement` Automated: [policy service](../../server/src/__tests__/company-skill-policy-service.test.ts) and [agent skills state](../../ui/src/pages/agent-skills/AgentSkillsTab.test.ts) cover policy/state contracts. @@ -81,6 +87,14 @@ version pins. The [concurrency regression](../../server/src/__tests__/company-sk checks shared work and separate actor access; the cache tests check that cleanup cannot remove a snapshot before the run stores its pins. +The [package storage regression](../../server/src/__tests__/skill-package-store.test.ts) +covers external snapshots, automatic legacy conversion and exact-commit recovery. A warm selected-repository +check must perform no object reads or repository-content SQL. Source/library +reconciliation remains available for discovery and mutable sources. The native context +regression checks that warm asset packaging reuses verified files, repairs missing/corrupt copies, +and notices source edits. Verify both writer-switch states during rollout; see +[package storage](../../doc/skill-package-storage.md). + ## Gotchas - The release picker requires `enableBetaSkills`. Do not invent an Instructions diff --git a/packages/db/src/migrations/0415_zippy_zemo.sql b/packages/db/src/migrations/0415_zippy_zemo.sql new file mode 100644 index 00000000000..86a052fc7a1 --- /dev/null +++ b/packages/db/src/migrations/0415_zippy_zemo.sql @@ -0,0 +1,81 @@ +CREATE TABLE IF NOT EXISTS "skill_package_deletions" ( + "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL, + "company_id" uuid NOT NULL, + "provider" text NOT NULL, + "archive_key" text NOT NULL, + "manifest_key" text NOT NULL, + "delete_after" timestamp with time zone NOT NULL +); +--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ALTER COLUMN "file_inventory" SET DEFAULT '[]'::jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "storage" jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "upload" jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "restore" jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "migration_at" timestamp with time zone DEFAULT now();--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "maintenance_at" timestamp with time zone DEFAULT now();--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "restore_retry_at" timestamp with time zone;--> statement-breakpoint +ALTER TABLE "company_skill_versions" ADD COLUMN IF NOT EXISTS "package_snapshot_id" uuid;--> statement-breakpoint +ALTER TABLE "company_skill_versions" ADD COLUMN IF NOT EXISTS "package_file_map" jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_versions" ADD COLUMN IF NOT EXISTS "migration_at" timestamp with time zone DEFAULT now();--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "skill_package_deletions_due_idx" ON "skill_package_deletions" USING btree ("delete_after","id");--> statement-breakpoint +DO $$ BEGIN +ALTER TABLE "company_skill_versions" ADD CONSTRAINT "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk" FOREIGN KEY ("package_snapshot_id") REFERENCES "public"."company_skill_repository_snapshots"("id") ON DELETE restrict ON UPDATE no action; +EXCEPTION WHEN duplicate_object THEN NULL; +END $$;--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_snapshots_migration_idx" ON "company_skill_repository_snapshots" USING btree ("migration_at") WHERE "company_skill_repository_snapshots"."migration_at" is not null;--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_snapshots_maintenance_idx" ON "company_skill_repository_snapshots" USING btree ("maintenance_at") WHERE "company_skill_repository_snapshots"."maintenance_at" is not null;--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_versions_migration_idx" ON "company_skill_versions" USING btree ("migration_at") WHERE "company_skill_versions"."migration_at" is not null;--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_versions_package_snapshot_idx" ON "company_skill_versions" USING btree ("package_snapshot_id");--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_versions_repository_snapshot_idx" ON "company_skill_versions" USING btree ("repository_snapshot_id"); +--> statement-breakpoint +CREATE OR REPLACE FUNCTION paperclip_queue_skill_package_deletion() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF OLD.storage IS NOT NULL AND (TG_OP = 'DELETE' OR OLD.storage->>'archiveKey' IS DISTINCT FROM NEW.storage->>'archiveKey') THEN + INSERT INTO skill_package_deletions(company_id, provider, archive_key, manifest_key, delete_after) + VALUES (OLD.company_id, OLD.storage->>'provider', OLD.storage->>'archiveKey', OLD.storage->>'manifestKey', now() + interval '30 days'); + END IF; + IF OLD.upload IS NOT NULL AND (TG_OP = 'DELETE' OR ( + OLD.upload->>'token' IS DISTINCT FROM NEW.upload->>'token' + AND OLD.upload->>'archiveKey' IS DISTINCT FROM NEW.storage->>'archiveKey' + )) THEN + INSERT INTO skill_package_deletions(company_id, provider, archive_key, manifest_key, delete_after) + VALUES (OLD.company_id, OLD.upload->>'provider', OLD.upload->>'archiveKey', OLD.upload->>'manifestKey', + greatest(now() + interval '1 hour', (OLD.upload->>'expiresAt')::timestamptz)); + END IF; + RETURN NULL; +END; +$$; +--> statement-breakpoint +DROP TRIGGER IF EXISTS skill_package_deletion_receipts ON company_skill_repository_snapshots; +--> statement-breakpoint +CREATE TRIGGER skill_package_deletion_receipts AFTER UPDATE OF storage, upload OR DELETE +ON company_skill_repository_snapshots FOR EACH ROW EXECUTE FUNCTION paperclip_queue_skill_package_deletion(); + +--> statement-breakpoint +CREATE OR REPLACE FUNCTION paperclip_schedule_skill_package_cleanup() RETURNS trigger LANGUAGE plpgsql AS $$ +DECLARE version_id uuid; +BEGIN + IF TG_TABLE_NAME = 'company_skills' THEN + version_id := OLD.current_version_id; + IF version_id IS NOT NULL AND (TG_OP = 'DELETE' OR version_id IS DISTINCT FROM NEW.current_version_id) THEN + UPDATE company_skill_repository_snapshots SET maintenance_at = now() + interval '30 days' + WHERE id IN (SELECT repository_snapshot_id FROM company_skill_versions WHERE id = version_id + UNION SELECT package_snapshot_id FROM company_skill_versions WHERE id = version_id); + END IF; + ELSE + UPDATE company_skill_repository_snapshots SET maintenance_at = now() + interval '1 day' + WHERE id = OLD.repository_snapshot_id OR id = OLD.package_snapshot_id; + END IF; + RETURN NULL; +END; +$$; +--> statement-breakpoint +DROP TRIGGER IF EXISTS skill_package_current_cleanup ON company_skills; +--> statement-breakpoint +CREATE TRIGGER skill_package_current_cleanup AFTER UPDATE OF current_version_id OR DELETE ON company_skills +FOR EACH ROW EXECUTE FUNCTION paperclip_schedule_skill_package_cleanup(); +--> statement-breakpoint +DROP TRIGGER IF EXISTS skill_package_version_cleanup ON company_skill_versions; +--> statement-breakpoint +CREATE TRIGGER skill_package_version_cleanup AFTER DELETE ON company_skill_versions +FOR EACH ROW EXECUTE FUNCTION paperclip_schedule_skill_package_cleanup(); diff --git a/packages/db/src/migrations/meta/0410_snapshot.json b/packages/db/src/migrations/meta/0415_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0410_snapshot.json rename to packages/db/src/migrations/meta/0415_snapshot.json index 4fb46f14289..d2c7322dea0 100644 --- a/packages/db/src/migrations/meta/0410_snapshot.json +++ b/packages/db/src/migrations/meta/0415_snapshot.json @@ -1,6 +1,6 @@ { - "id": "c02426e4-de1a-4ce3-a541-99bb970c272a", - "prevId": "94acf3fe-50f3-4a97-a7fe-865abbc9d0a3", + "id": "dce41524-29a3-4552-ba5d-f0abefbf1ea1", + "prevId": "27c9008c-60e3-449b-9630-9e4977fad444", "version": "7", "dialect": "postgresql", "tables": { @@ -1351,6 +1351,39 @@ "method": "btree", "with": {} }, + "activity_log_company_entity_created_idx": { + "name": "activity_log_company_entity_created_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_type", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "entity_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "activity_log_company_user_issue_actor_idx": { "name": "activity_log_company_user_issue_actor_idx", "columns": [ @@ -19046,7 +19079,46 @@ "name": "file_inventory", "type": "jsonb", "primaryKey": false, - "notNull": true + "notNull": true, + "default": "'[]'::jsonb" + }, + "storage": { + "name": "storage", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "upload": { + "name": "upload", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "restore": { + "name": "restore", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "migration_at": { + "name": "migration_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false, + "default": "now()" + }, + "maintenance_at": { + "name": "maintenance_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false, + "default": "now()" + }, + "restore_retry_at": { + "name": "restore_retry_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false }, "created_at": { "name": "created_at", @@ -19077,6 +19149,38 @@ "concurrently": false, "method": "btree", "with": {} + }, + "company_skill_snapshots_migration_idx": { + "name": "company_skill_snapshots_migration_idx", + "columns": [ + { + "expression": "migration_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"company_skill_repository_snapshots\".\"migration_at\" is not null", + "concurrently": false, + "method": "btree", + "with": {} + }, + "company_skill_snapshots_maintenance_idx": { + "name": "company_skill_snapshots_maintenance_idx", + "columns": [ + { + "expression": "maintenance_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"company_skill_repository_snapshots\".\"maintenance_at\" is not null", + "concurrently": false, + "method": "btree", + "with": {} } }, "foreignKeys": { @@ -19966,6 +20070,25 @@ "primaryKey": false, "notNull": false }, + "package_snapshot_id": { + "name": "package_snapshot_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "package_file_map": { + "name": "package_file_map", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "migration_at": { + "name": "migration_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false, + "default": "now()" + }, "revision_number": { "name": "revision_number", "type": "integer", @@ -20024,6 +20147,52 @@ } }, "indexes": { + "company_skill_versions_migration_idx": { + "name": "company_skill_versions_migration_idx", + "columns": [ + { + "expression": "migration_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"company_skill_versions\".\"migration_at\" is not null", + "concurrently": false, + "method": "btree", + "with": {} + }, + "company_skill_versions_package_snapshot_idx": { + "name": "company_skill_versions_package_snapshot_idx", + "columns": [ + { + "expression": "package_snapshot_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "company_skill_versions_repository_snapshot_idx": { + "name": "company_skill_versions_repository_snapshot_idx", + "columns": [ + { + "expression": "repository_snapshot_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "company_skill_versions_skill_revision_idx": { "name": "company_skill_versions_skill_revision_idx", "columns": [ @@ -20135,6 +20304,19 @@ "onDelete": "restrict", "onUpdate": "no action" }, + "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk": { + "name": "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk", + "tableFrom": "company_skill_versions", + "tableTo": "company_skill_repository_snapshots", + "columnsFrom": [ + "package_snapshot_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "restrict", + "onUpdate": "no action" + }, "company_skill_versions_author_agent_id_agents_id_fk": { "name": "company_skill_versions_author_agent_id_agents_id_fk", "tableFrom": "company_skill_versions", @@ -20580,6 +20762,78 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.skill_package_deletions": { + "name": "skill_package_deletions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "archive_key": { + "name": "archive_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "manifest_key": { + "name": "manifest_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "delete_after": { + "name": "delete_after", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "skill_package_deletions_due_idx": { + "name": "skill_package_deletions_due_idx", + "columns": [ + { + "expression": "delete_after", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.company_transfer_runs": { "name": "company_transfer_runs", "schema": "", @@ -33771,6 +34025,34 @@ } }, "indexes": { + "heartbeat_runs_native_unpresented_response_idx": { + "name": "heartbeat_runs_native_unpresented_response_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "native_issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" in ('succeeded', 'failed', 'timed_out') AND \"heartbeat_runs\".\"finished_at\" is not null AND jsonb_typeof(\"heartbeat_runs\".\"result_json\" #> '{presentationDecision,commentId}') is distinct from 'string'", + "concurrently": false, + "method": "btree", + "with": {} + }, "heartbeat_runs_cancelled_workspace_preservation_idx": { "name": "heartbeat_runs_cancelled_workspace_preservation_idx", "columns": [ @@ -36475,6 +36757,40 @@ "method": "btree", "with": {} }, + "issue_comments_review_response_receipt_idx": { + "name": "issue_comments_review_response_receipt_idx", + "columns": [ + { + "expression": "created_by_run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "author_agent_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"issue_comments\".\"author_type\" = 'agent' AND \"issue_comments\".\"metadata\"->>'authorizationReason' = 'allow_chat_run_presentation'", + "concurrently": false, + "method": "btree", + "with": {} + }, "issue_comments_company_idx": { "name": "issue_comments_company_idx", "columns": [ @@ -43108,6 +43424,34 @@ } }, "indexes": { + "native_run_finalizations_pending_effect_owner_idx": { + "name": "native_run_finalizations_pending_effect_owner_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "decision_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"native_run_finalizations\".\"decision_id\" is not null AND \"native_run_finalizations\".\"result_id\" is not null AND \"native_run_finalizations\".\"phase\" <> 'terminal_failure'", + "concurrently": false, + "method": "btree", + "with": {} + }, "native_run_finalizations_unsafe_export_recovery_idx": { "name": "native_run_finalizations_unsafe_export_recovery_idx", "columns": [ @@ -52440,6 +52784,34 @@ } }, "indexes": { + "status_decision_effects_pending_delivery_idx": { + "name": "status_decision_effects_pending_delivery_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "decision_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"status_decision_effects\".\"delivery_state\" in ('pending', 'failed')", + "concurrently": false, + "method": "btree", + "with": {} + }, "status_decision_effects_decision_ordinal_uq": { "name": "status_decision_effects_decision_ordinal_uq", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 6fdfc546aee..e637a1f7a26 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2892,6 +2892,13 @@ "when": 1793280975229, "tag": "0414_closed_edwin_jarvis", "breakpoints": true + }, + { + "idx": 415, + "version": "7", + "when": 1793289638018, + "tag": "0415_zippy_zemo", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/company_skills.ts b/packages/db/src/schema/company_skills.ts index d6ce60b2aeb..4f4fc2f0e6e 100644 --- a/packages/db/src/schema/company_skills.ts +++ b/packages/db/src/schema/company_skills.ts @@ -70,14 +70,78 @@ export type CompanySkillVersionFileInventoryEntry = CompanySkillFileInventoryEnt executable?: boolean; }; +export type SkillPackageStorageReceipt = { + durable?: boolean; + format: "tar-gzip-v1"; + provider: "local_disk" | "s3"; + archiveKey: string; + archiveSha256: string; + archiveBytes: number; + manifestKey: string; + manifestSha256: string; + manifestBytes: number; + fileCount: number; + totalBytes: number; +}; + +export type SkillPackageUpload = { + token: string; + legacy?: boolean; + provider: "local_disk" | "s3"; + archiveKey: string; + manifestKey: string; + expiresAt: string; +}; + +/** Exact input to the v1 packager. File bodies never live in this recipe. */ +export type SkillPackageRestoreRecipe = { + version: 1; + sourceId: string; + repositoryId: string; + repositoryUrl: string; + commitSha: string; + skillPath: string | null; + manifest: { + format: "paperclip-skill-package/v1"; + contentHash: string; + totalBytes: number; + files: Array<{ path: string; kind: CompanySkillFileInventoryEntry["kind"]; encoding: "utf8" | "base64"; + executable: boolean; size: number; sha256: string }>; + }; +}; + +export type SkillPackageFileMapping = CompanySkillFileInventoryEntry & { sourcePath: string; encoding?: "utf8" | "base64" }; + +/** Durable deletion receipts survive company/snapshot deletion until storage confirms removal. */ +export const skillPackageDeletions = pgTable("skill_package_deletions", { + id: uuid("id").primaryKey().defaultRandom(), + companyId: uuid("company_id").notNull(), + provider: text("provider").notNull(), + archiveKey: text("archive_key").notNull(), + manifestKey: text("manifest_key").notNull(), + deleteAfter: timestamp("delete_after", { withTimezone: true }).notNull(), +}, table => ({ due: index("skill_package_deletions_due_idx").on(table.deleteAfter, table.id) })); + /** Immutable repository bytes shared by all skill versions from the same package. */ export const companySkillRepositorySnapshots = pgTable("company_skill_repository_snapshots", { id: uuid("id").primaryKey().defaultRandom(), companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }), contentHash: text("content_hash").notNull(), - fileInventory: jsonb("file_inventory").$type().notNull(), + // Legacy payload only. New packages retain bytes in the configured storage backend. + fileInventory: jsonb("file_inventory").$type().notNull().default([]), + storage: jsonb("storage").$type(), + upload: jsonb("upload").$type(), + restore: jsonb("restore").$type(), + // These deadlines are consumed by the delivery coordinator, never polled per turn. + migrationAt: timestamp("migration_at", { withTimezone: true }).defaultNow(), + maintenanceAt: timestamp("maintenance_at", { withTimezone: true }).defaultNow(), + restoreRetryAt: timestamp("restore_retry_at", { withTimezone: true }), createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), -}, table => ({ content: uniqueIndex("company_skill_repository_snapshots_content_idx").on(table.companyId, table.contentHash) })); +}, table => ({ + content: uniqueIndex("company_skill_repository_snapshots_content_idx").on(table.companyId, table.contentHash), + migration: index("company_skill_snapshots_migration_idx").on(table.migrationAt).where(sql`${table.migrationAt} is not null`), + maintenance: index("company_skill_snapshots_maintenance_idx").on(table.maintenanceAt).where(sql`${table.maintenanceAt} is not null`), +})); export const companySkillVersions = pgTable( "company_skill_versions", @@ -87,6 +151,9 @@ export const companySkillVersions = pgTable( companySkillId: uuid("company_skill_id").notNull().references(() => companySkills.id, { onDelete: "cascade" }), repositorySnapshotId: uuid("repository_snapshot_id").references(() => companySkillRepositorySnapshots.id, { onDelete: "restrict" }), repositorySkillPath: text("repository_skill_path"), + packageSnapshotId: uuid("package_snapshot_id").references(() => companySkillRepositorySnapshots.id, { onDelete: "restrict" }), + packageFileMap: jsonb("package_file_map").$type(), + migrationAt: timestamp("migration_at", { withTimezone: true }).defaultNow(), revisionNumber: integer("revision_number").notNull(), label: text("label"), releaseId: text("release_id"), @@ -98,6 +165,9 @@ export const companySkillVersions = pgTable( createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), }, (table) => ({ + migrationIdx: index("company_skill_versions_migration_idx").on(table.migrationAt).where(sql`${table.migrationAt} is not null`), + packageSnapshotIdx: index("company_skill_versions_package_snapshot_idx").on(table.packageSnapshotId), + repositorySnapshotIdx: index("company_skill_versions_repository_snapshot_idx").on(table.repositorySnapshotId), companySkillRevisionUniqueIdx: uniqueIndex("company_skill_versions_skill_revision_idx").on( table.companySkillId, table.revisionNumber, diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index 33d9abd518b..2e7e342dad3 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -193,13 +193,14 @@ export { } from "./tool_access.js"; export { companySkills, - companySkillVersions, companySkillRepositorySnapshots, + companySkillVersions, companySkillRepositorySnapshots, skillPackageDeletions, companySkillStars, companySkillComments, companySkillTestInputs, companySkillTestRunTemplates, companySkillTestRuns, } from "./company_skills.js"; +export type { SkillPackageStorageReceipt, SkillPackageUpload, SkillPackageRestoreRecipe, SkillPackageFileMapping } from "./company_skills.js"; export { plugins } from "./plugins.js"; export { pluginConfig } from "./plugin_config.js"; export { pluginCompanySettings } from "./plugin_company_settings.js"; diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index fb7f6b2e31a..77c4479d15c 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -787,6 +787,7 @@ export type { CompanySkillListQuery, CompanySkillCategoryCount, CompanySkillVersion, + CompanySkillVersionSummary, CompanySkillVersionCreateRequest, CompanySkillStarResult, CompanySkillComment, diff --git a/packages/shared/src/types/company-skill.ts b/packages/shared/src/types/company-skill.ts index 9ab84634f21..0f24c5646b5 100644 --- a/packages/shared/src/types/company-skill.ts +++ b/packages/shared/src/types/company-skill.ts @@ -138,7 +138,7 @@ export interface CompanySkillDetail extends CompanySkill { sourceLabel: string | null; sourceBadge: CompanySkillSourceBadge; sourcePath: string | null; - currentVersion: CompanySkillVersion | null; + currentVersion: CompanySkillVersionSummary | null; starredByCurrentActor: boolean; } @@ -174,6 +174,11 @@ export interface CompanySkillVersion { createdAt: Date; } +/** Version listings carry file names/kinds; fetch one version to read its bytes. */ +export type CompanySkillVersionSummary = Omit & { + fileInventory: CompanySkillFileInventoryEntry[]; +}; + export interface CompanySkillVersionCreateRequest { label?: string | null; } @@ -608,7 +613,7 @@ export interface CompanySkillTestRunHarnessContent { } export interface CompanySkillTestRunDetail extends CompanySkillTestRun { - skillVersion: CompanySkillVersion; + skillVersion: CompanySkillVersionSummary; outputBody: string; harnessContent: CompanySkillTestRunHarnessContent; harnessIssue: { diff --git a/packages/shared/src/types/index.ts b/packages/shared/src/types/index.ts index 930bd5786cf..5feb5cdaad2 100644 --- a/packages/shared/src/types/index.ts +++ b/packages/shared/src/types/index.ts @@ -171,6 +171,7 @@ export type { CompanySkillListQuery, CompanySkillCategoryCount, CompanySkillVersion, + CompanySkillVersionSummary, CompanySkillVersionCreateRequest, CompanySkillStarResult, CompanySkillComment, diff --git a/server/package.json b/server/package.json index 46f9beff067..0265e98d04e 100644 --- a/server/package.json +++ b/server/package.json @@ -115,6 +115,7 @@ "smol-toml": "^1.9.1", "ssh2": "^1.17.0", "svix": "1.76.1", + "tar-stream": "3.2.0", "ws": "^8.22.0", "zod": "^4.4.3", "@opentelemetry/resources": "2.10.0" @@ -142,6 +143,7 @@ "@types/node": "^24.0.0", "@types/sharp": "^0.32.0", "@types/supertest": "^7.2.1", + "@types/tar-stream": "^3.1.5", "@types/ws": "^8.18.2", "cross-env": "^10.1.0", "supertest": "^7.0.0", diff --git a/server/scripts/benchmark-native-skill-preparation.ts b/server/scripts/benchmark-native-skill-preparation.ts index da1310afdb0..c4897afaf3e 100644 --- a/server/scripts/benchmark-native-skill-preparation.ts +++ b/server/scripts/benchmark-native-skill-preparation.ts @@ -18,11 +18,20 @@ import { authUsers, companyMemberships, companySkillSources, heartbeatRuns, proj import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; import { createRunPreparationTiming, withRunExecutionTiming } from "../src/services/execution-preparation-timing.js"; import { repositorySkillProjection } from "../src/services/skill-repository-package.js"; +import { materializeSelectedSkills } from "../src/services/native-runtime/runtime-context.js"; +import { skillPackageStore } from "../src/services/skill-package-store.js"; +import { getStorageService } from "../src/storage/index.js"; import { skillFileBytes, skillSnapshotHash } from "../src/services/skill-snapshot.js"; const argument = (name: string) => process.argv.find(value => value.startsWith(`${name}=`))?.slice(name.length + 1); const exportOtlp = argument("--export-otlp") === "true"; const authenticated = argument("--actor") === "agent"; +const warmRuns = Number(argument("--warm-runs") ?? 2); +assert.ok(Number.isSafeInteger(warmRuns) && warmRuns >= 1 && warmRuns <= 100); +const selectedSkillCount = Number(argument("--selected-skills") ?? 1); +assert.ok(Number.isSafeInteger(selectedSkillCount) && selectedSkillCount >= 1 && selectedSkillCount <= 100); +const nativeAssets = argument("--native-assets") === "true"; +const externalPackages = argument("--package-store") === "external"; const stagingShape = argument("--staging-shape") === "true"; // This standalone benchmark owns its tracer. Load the service graph with the // app-wide SDK disabled, then restore the operator's export configuration. @@ -44,7 +53,9 @@ const sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { }).trim(); const sourceHash = createHash("sha256"); for (const file of [import.meta.url, new URL("../src/services/company-skills.ts", import.meta.url), - new URL("../src/services/skill-version-cache.ts", import.meta.url), new URL("../src/services/execution-preparation-timing.ts", import.meta.url)]) { + new URL("../src/services/skill-version-cache.ts", import.meta.url), new URL("../src/services/execution-preparation-timing.ts", import.meta.url), + ...["skill-package-store", "skill-package-runtime", "skill-package-files", "skill-package-cache", "file-manifest-cache", + "skill-package-recovery", "skill-package-work", "native-runtime/runtime-context", "native-runtime/skill-asset-cache", "native-runtime/runtime-asset-retention"].map(name => new URL(`../src/services/${name}.ts`, import.meta.url))]) { sourceHash.update(await fs.readFile(new URL(file))); } const sourceChangesHash = sourceHash.digest("hex"); @@ -55,6 +66,14 @@ const files: CompanySkillVersionFileInventoryEntry[] = snapshotFile ? JSON.parse(await fs.readFile(snapshotFile, "utf8")) : [{ path: entrypoint, kind: "skill", content: "---\nname: diagnostic\ndescription: Native preparation benchmark\n---\nReport the model ID.\n" }, ...Array.from({ length: 207 }, (_, index) => ({ path: `assets/${index}.txt`, kind: "reference" as const, content: "x".repeat(52_000) }))]; +const entrypoints = [entrypoint]; +for (let i = 1; i < selectedSkillCount; i++) { + const source = files.find(file => file.path === entrypoint)!; + const next = `skills/diagnostic-${i}/SKILL.md`; + assert.ok(!files.some(file => file.path === next)); + files.push({ ...source, path: next, content: source.content.replace("name: diagnostic", `name: diagnostic-${i}`) }); + entrypoints.push(next); +} const projected = repositorySkillProjection(files, entrypoint); const previous = { home: process.env.PAPERCLIP_HOME, instance: process.env.PAPERCLIP_INSTANCE_ID, endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }; const home = await fs.mkdtemp(path.join(os.tmpdir(), "native-skill-benchmark-")); @@ -113,12 +132,15 @@ try { catch (error) { await cleanup(); throw error; } return { connectionString, cleanup }; } + console.error("benchmark: starting isolated database"); database = await startDatabase(); + console.error("benchmark: seeding isolated fixture"); const networkDelayMs = Number(argument("--network-delay-ms") ?? 0); assert.ok(Number.isFinite(networkDelayMs) && networkDelayMs >= 0 && networkDelayMs <= 100); const target = new URL(database.connectionString); const upstreamPort = Number(target.port); - proxy = networkDelayMs ? createServer(client => { + let sqlResponseBytes = 0; + proxy = createServer(client => { const upstream = connect({ host: "127.0.0.1", port: upstreamPort }); for (const socket of [client, upstream]) { socket.setNoDelay(true); @@ -126,6 +148,7 @@ try { socket.on("error", () => { client.destroy(); upstream.destroy(); }); socket.on("close", () => { connections.delete(socket); }); } + upstream.on("data", chunk => { sqlResponseBytes += chunk.length; }); const relay = (source: Socket, destination: Socket) => { let pendingBytes = 0; source.on("data", chunk => { @@ -142,7 +165,7 @@ try { source.on("end", () => setTimeout(() => destination.end(), networkDelayMs)); }; relay(client, upstream); relay(upstream, client); - }) : null; + }); if (proxy) { const server = proxy; await new Promise((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); }); @@ -154,8 +177,7 @@ try { const companyId = randomUUID(); const agentId = randomUUID(); const userId = randomUUID(); - const skillId = randomUUID(); - const key = `company/${companyId}/diagnostic`; + const selections = new Map(); const samples: unknown[] = []; let rootPoolQueries = 0; db.$client.options.debug = () => { rootPoolQueries++; }; @@ -188,27 +210,50 @@ try { }))); await db.insert(companySkills).values({ companyId, key: "fixture/catalog", slug: "catalog", name: "Unselected catalog", sourceType: "catalog", markdown: "x".repeat(1_578) }); } - await db.insert(companySkills).values({ id: skillId, companyId, key, slug: "diagnostic", name: "Diagnostic", - markdown: files.find(file => file.path === entrypoint)!.content, sourceType: "github", - sourceLocator: "https://example.invalid/immutable-fixture", metadata: { packageMode: "repository", snapshotHash: "fixture" } }); - const [snapshot] = await db.insert(companySkillRepositorySnapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); - const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, revisionNumber: 1, - repositorySnapshotId: snapshot!.id, repositorySkillPath: entrypoint }).returning(); - await db.update(companySkills).set({ currentVersionId: version!.id }).where(eq(companySkills.id, skillId)); + const snapshotId = externalPackages ? await skillPackageStore(db).prepare(companyId, files) + : (await db.insert(companySkillRepositorySnapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning())[0]!.id; + let objectGets = 0, objectReadBytes = 0; + const backend = getStorageService(); + const getObject = backend.getObject.bind(backend); + backend.getObject = async (...args) => { + const object = await getObject(...args); + objectGets++; objectReadBytes += object.contentLength ?? 0; + return object; + }; + for (const [index, selectedPath] of entrypoints.entries()) { + const skillId = randomUUID(); + const key = `company/${companyId}/diagnostic-${index}`; + await db.insert(companySkills).values({ id: skillId, companyId, key, slug: `diagnostic-${index}`, name: `Diagnostic ${index}`, + markdown: files.find(file => file.path === selectedPath)!.content, sourceType: "github", + sourceLocator: "https://example.invalid/immutable-fixture", metadata: { packageMode: "repository", snapshotHash: "fixture" } }); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, revisionNumber: 1, + repositorySnapshotId: snapshotId, repositorySkillPath: selectedPath }).returning(); + await db.update(companySkills).set({ currentVersionId: version!.id }).where(eq(companySkills.id, skillId)); + selections.set(key, version!.id); + } const benchmarkDb = db; const service = companySkillService(db); - for (const label of ["cold", "warm-1", "warm-2", "warm-new-service"]) { + console.error("benchmark: measuring preparations"); + for (const label of ["cold", ...Array.from({ length: warmRuns }, (_, index) => `warm-${index + 1}`), "warm-new-service"]) { exporter.reset(); rootPoolQueries = 0; const selected: ReturnType = label === "warm-new-service" ? companySkillService(db) : service; const runId = randomUUID(); if (authenticated) await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", responsibleUserId: userId, contextSnapshot: { executionPolicy: {} } }); rootPoolQueries = 0; + sqlResponseBytes = 0; objectGets = 0; objectReadBytes = 0; + let nativeAssetMs = 0; + const cpuStarted = process.cpuUsage(); const measuredStarted = performance.now(); const resolve = () => selected.withRuntimeSkillEntries(companyId, { - onlyKeys: new Set([key]), versionSelections: new Map([[key, version!.id]]), snapshotLatest: true, + onlyKeys: new Set(selections.keys()), versionSelections: selections, snapshotLatest: true, }, async entries => { - if (authenticated) await benchmarkDb.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ skillVersions: { [key]: version!.id }, skillSelectionSnapshot: true })}::jsonb` }).where(eq(heartbeatRuns.id, runId)); + if (authenticated) await benchmarkDb.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ skillVersions: Object.fromEntries(selections), skillSelectionSnapshot: true })}::jsonb` }).where(eq(heartbeatRuns.id, runId)); + if (nativeAssets) { + const started = performance.now(); + await materializeSelectedSkills({ paperclipSkillSync: { desiredSkills: [...selections.keys()] } }, entries, true); + nativeAssetMs = performance.now() - started; + } return entries; }); const entries: Awaited> = await withRunExecutionTiming(runId, () => createRunPreparationTiming(runId)("configuration_skills", () => authenticated @@ -216,11 +261,15 @@ try { : resolve())); const elapsedMs = performance.now() - measuredStarted; const queryCount = rootPoolQueries; - assert.equal(entries.length, 1); + const cpu = process.cpuUsage(cpuStarted); + const measuredIo = { nativeAssetMs, sqlResponseBytes, objectGets, objectReadBytes, rssBytes: process.memoryUsage().rss, + cpuUserMs: cpu.user / 1000, cpuSystemMs: cpu.system / 1000, processPeakRssBytes: process.resourceUsage().maxRSS * 1024 }; + assert.equal(entries.length, selectedSkillCount); + for (const entry of entries) assert.equal(entry.sourceStatus, "available", entry.missingDetail ?? "Skill unavailable"); assert.equal(entries[0]!.sourceStatus, "available", entries[0]!.missingDetail ?? "Skill unavailable"); // Verify the whole delivered repository, not just its authored SKILL.md. for (const file of projected) { - const location = path.join(entries[0]!.source, file.path); + const location = path.join(entries.find(entry => entry.key === selections.keys().next().value)!.source, file.path); assert.deepEqual(await fs.readFile(location), skillFileBytes(file)); if (process.platform !== "win32") assert.equal(Boolean((await fs.stat(location)).mode & 0o111), Boolean(file.executable)); } @@ -230,17 +279,32 @@ try { assert.ok(spans.some(span => span.name === "skills.version.verify" && span.attributes["paperclip.execution.fileCount"] === projected.length)); assert.equal(new Set(spans.map(span => span.spanContext().traceId)).size, 1); assert.ok(spans.every(span => Object.keys(span.attributes).every(key => ["paperclip.task.run.run_id", "paperclip.execution.fileCount", "paperclip.execution.bytes"].includes(key)))); - samples.push({ label, elapsedMs, rootPoolQueries: queryCount, traceId: spans[0]!.spanContext().traceId, + samples.push({ label, elapsedMs, ...measuredIo, rootPoolQueries: queryCount, traceId: spans[0]!.spanContext().traceId, phases: spans.filter(span => span.name.startsWith("skills.")).map(span => ({ name: span.name, durationMs: span.duration[0] * 1000 + span.duration[1] / 1e6, spanId: span.spanContext().spanId, parentSpanId: span.parentSpanContext?.spanId, attributes: span.attributes })) }); - console.error(JSON.stringify({ label, elapsedMs, rootPoolQueries: queryCount })); + console.error(JSON.stringify({ label, elapsedMs, ...measuredIo, rootPoolQueries: queryCount })); } + async function diskUsage(root: string): Promise<{ allocatedBytes: number; fileBytes: number; files: number }> { + const total = { allocatedBytes: 0, fileBytes: 0, files: 0 }; + for (const item of await fs.readdir(root, { withFileTypes: true })) { + const target = path.join(root, item.name); + if (item.isDirectory()) { + const child = await diskUsage(target); + total.allocatedBytes += child.allocatedBytes; total.fileBytes += child.fileBytes; total.files += child.files; + } else if (item.isFile()) { + const stat = await fs.stat(target); + total.allocatedBytes += stat.blocks * 512; total.fileBytes += stat.size; total.files++; + } + } + return total; + } + const instanceDisk = await diskUsage(home); console.log(JSON.stringify({ schema: "paperclip.native-skill-preparation-benchmark.v1", sourceRevision, sourceChangesHash, exportedOtlp: exportOtlp, source: snapshotFile ? "supplied immutable snapshot" : "synthetic repository", - networkDelayMsPerDirection: networkDelayMs, authenticated, stagingShape, + networkDelayMsPerDirection: networkDelayMs, authenticated, stagingShape, externalPackages, nativeAssets, selectedSkillCount, instanceDisk, repositoryHash: skillSnapshotHash(files), repositoryFiles: files.length, runtimeFiles: projected.length, runtimeBytes: projected.reduce((sum, file) => sum + skillFileBytes(file).length, 0), samples }, null, 2)); } finally { @@ -256,5 +320,12 @@ try { for (const [name, value] of Object.entries({ PAPERCLIP_HOME: previous.home, PAPERCLIP_INSTANCE_ID: previous.instance, OTEL_EXPORTER_OTLP_ENDPOINT: previous.endpoint })) { if (value === undefined) delete process.env[name]; else process.env[name] = value; } + async function writableDirectories(root: string): Promise { + const stat = await fs.lstat(root); + if (!stat.isDirectory() || stat.isSymbolicLink()) return; + await fs.chmod(root, 0o700); + for (const child of await fs.readdir(root, { withFileTypes: true })) if (child.isDirectory()) await writableDirectories(path.join(root, child.name)); + } + await writableDirectories(home); await fs.rm(home, { recursive: true, force: true }); } diff --git a/server/scripts/migrate-skill-packages.ts b/server/scripts/migrate-skill-packages.ts new file mode 100644 index 00000000000..1f089f430c4 --- /dev/null +++ b/server/scripts/migrate-skill-packages.ts @@ -0,0 +1,21 @@ +/** Run from the server package with its configured storage and DATABASE_URL. */ +import { createDb } from "@paperclipai/db"; +import { skillPackageMaintenance } from "../src/services/skill-package-maintenance.js"; + +const value = (flag: string) => process.argv.find(arg => arg.startsWith(`${flag}=`))?.slice(flag.length + 1); +const phase = value("--phase") ?? "snapshots"; +const url = process.env.DATABASE_URL; +if (!url) throw new Error("DATABASE_URL must explicitly select the database to migrate"); +const limit = Number(value("--limit") ?? 10); +if (!Number.isSafeInteger(limit) || limit < 1 || limit > 50) throw new Error("--limit must be between 1 and 50"); +const db = createDb(url, { maxConnections: 2 }); +try { + const service = skillPackageMaintenance(db); + const actions = { snapshots: service.migrateSnapshots, versions: service.migrateVersions, + orphans: service.collectOrphans, deletions: service.deleteObjects }; + const action = actions[phase as keyof typeof actions]; + if (!action) throw new Error("--phase must be snapshots, versions, orphans, or deletions"); + const apply = process.argv.includes("--apply"); + const result = await action({ apply, clearLegacy: process.argv.includes("--clear-legacy"), limit, after: value("--after"), companyId: value("--company") }); + console.log(JSON.stringify({ phase, apply, ...result })); +} finally { await db.$client.end({ timeout: 1 }); } diff --git a/server/scripts/verify-skill-package-s3.ts b/server/scripts/verify-skill-package-s3.ts new file mode 100644 index 00000000000..aaefcc4186b --- /dev/null +++ b/server/scripts/verify-skill-package-s3.ts @@ -0,0 +1,59 @@ +/** Disposable, loopback-only S3 compatibility check. No application data is used. */ +import assert from "node:assert/strict"; +import { randomBytes, randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { S3Client, CreateBucketCommand, DeleteBucketCommand } from "@aws-sdk/client-s3"; +import { companies, createDb, startEmbeddedPostgresTestDatabase } from "@paperclipai/db"; +import { createStorageService } from "../src/storage/service.js"; +import { createS3StorageProvider } from "../src/storage/s3-provider.js"; +import { skillPackagePaths, skillPackageStore } from "../src/services/skill-package-store.js"; +import { skillSnapshotHash } from "../src/services/skill-snapshot.js"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; + +const endpoint = process.env.PAPERCLIP_SKILL_PACKAGE_TEST_S3_ENDPOINT; +assert.ok(endpoint, "Set PAPERCLIP_SKILL_PACKAGE_TEST_S3_ENDPOINT to a disposable loopback S3 server"); +assert.ok(["localhost", "127.0.0.1", "[::1]"].includes(new URL(endpoint).hostname), "Only loopback test storage is allowed"); +const region = "us-east-1", bucket = `skill-package-test-${randomUUID()}`; +const client = new S3Client({ region, endpoint, forcePathStyle: true }); +const database = await startEmbeddedPostgresTestDatabase("skill-package-s3-"); +const db = createDb(database.connectionString); +const home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-package-s3-")); +const previousHome = process.env.PAPERCLIP_HOME; +process.env.PAPERCLIP_HOME = home; +const storage = createStorageService(createS3StorageProvider({ bucket, region, endpoint, forcePathStyle: true })); +const companyId = randomUUID(); +const keys: string[] = []; +let bucketCreated = false; +try { + await client.send(new CreateBucketCommand({ Bucket: bucket })); + bucketCreated = true; + await db.insert(companies).values({ id: companyId, name: "Disposable S3 check", issuePrefix: "S3T" }); + const files: CompanySkillVersionFileInventoryEntry[] = [ + { path: "SKILL.md", kind: "skill", content: "---\nname: s3-check\ndescription: Disposable compatibility check\n---\nTest only.\n" }, + ...Array.from({ length: 20 }, (_, index) => ({ path: `assets/${index}.bin`, kind: "asset" as const, + content: randomBytes(900_000).toString("base64"), encoding: "base64" as const })), + { path: "scripts/check.sh", kind: "script", content: "#!/bin/sh\nexit 0\n", executable: true }, + ]; + const store = skillPackageStore(db, storage); + const id = await store.prepare(companyId, files); + const receipt = (await store.descriptor(companyId, id)).storage!; + keys.push(receipt.archiveKey, receipt.manifestKey); + assert.ok(receipt.archiveBytes > 16 * 1024 * 1024, "Exercise at least three multipart upload parts"); + assert.equal(skillSnapshotHash(await store.files(companyId, id)), skillSnapshotHash(files)); + await fs.rm(skillPackagePaths(companyId, id).directory, { recursive: true }); + assert.equal(skillSnapshotHash(await store.files(companyId, id)), skillSnapshotHash(files)); + assert.equal(await store.verifyDurable(companyId, id), true); + console.log(JSON.stringify({ provider: "s3", files: files.length, archiveBytes: receipt.archiveBytes, + decodedBytes: receipt.totalBytes, multipartRoundTrip: true, coldRestore: true, verifiedDurable: true })); +} finally { + await Promise.allSettled(keys.map(key => storage.deleteObject(companyId, key))); + if (bucketCreated) await client.send(new DeleteBucketCommand({ Bucket: bucket })).catch(error => console.error("Disposable bucket cleanup failed", error)); + client.destroy(); + await db.$client.end({ timeout: 1 }); + await database.cleanup(); + await fs.rm(home, { recursive: true, force: true }); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; +} diff --git a/server/src/__tests__/company-skills-service.test.ts b/server/src/__tests__/company-skills-service.test.ts index 08961fe7ec6..a88634af5e5 100644 --- a/server/src/__tests__/company-skills-service.test.ts +++ b/server/src/__tests__/company-skills-service.test.ts @@ -53,7 +53,9 @@ describeEmbeddedPostgres("companySkillService.list", () => { return skillDir; } + const previousPackageStorage = process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; beforeAll(async () => { + process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "true"; tempDb = await startEmbeddedPostgresTestDatabase("paperclip-company-skills-service-"); oldPaperclipHome = process.env.PAPERCLIP_HOME; oldPaperclipInstanceId = process.env.PAPERCLIP_INSTANCE_ID; @@ -80,6 +82,8 @@ describeEmbeddedPostgres("companySkillService.list", () => { }); afterAll(async () => { + if (previousPackageStorage === undefined) delete process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; + else process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = previousPackageStorage; if (oldPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = oldPaperclipHome; if (oldPaperclipInstanceId === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; @@ -233,7 +237,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { await db.insert(companies).values({ id: companyId, name: "Rollback retry", issuePrefix: `T${companyId.slice(0, 6)}` }); const input = { name: "Retryable", slug: "retryable", markdown: "# durable bytes\n" }; await expect(db.transaction(async (tx) => { - await companySkillService(tx as any).createLocalSkill(companyId, input); + await companySkillService(tx as any, { packageDb: db }).createLocalSkill(companyId, input); throw new Error("simulate outer rollback"); })).rejects.toThrow("simulate outer rollback"); @@ -383,7 +387,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { await db.insert(companies).values({ id: companyId, name: "Rollback conflict", issuePrefix: `T${companyId.slice(0, 6)}` }); const original = { name: "Original", slug: "rollback-conflict", markdown: "# original bytes\n" }; await expect(db.transaction(async (tx) => { - await companySkillService(tx as any).createLocalSkill(companyId, original); + await companySkillService(tx as any, { packageDb: db }).createLocalSkill(companyId, original); throw new Error("simulate outer rollback"); })).rejects.toThrow("simulate outer rollback"); @@ -894,7 +898,8 @@ describeEmbeddedPostgres("companySkillService.list", () => { releasedAt: new Date("2026-07-21T00:00:00.000Z"), }); if (!champion) throw new Error("Expected seeded v7-roster release"); - const championHashes = Object.fromEntries(champion.fileInventory.map((entry) => [ + const championVersion = (await svc.getVersion(companyId, paperclipSkill.id, champion.id))!; + const championHashes = Object.fromEntries(championVersion.fileInventory.map((entry) => [ entry.path, createHash("sha256").update(entry.content).digest("hex"), ])); @@ -2701,7 +2706,8 @@ describeEmbeddedPostgres("companySkillService.list", () => { versions = await svc.listVersions(companyId, skill.id); expect(versions).toHaveLength(2); expect(versions[0]).toMatchObject({ revisionNumber: 2, authorUserId: "board" }); - expect(versions[0]!.fileInventory).toEqual([ + expect(versions[0]!.fileInventory).toEqual([{ path: "SKILL.md", kind: "skill" }]); + expect((await svc.getVersion(companyId, skill.id, versions[0]!.id))!.fileInventory).toEqual([ expect.objectContaining({ path: "SKILL.md", content: editedMarkdown }), ]); await expect(svc.getById(companyId, skill.id)).resolves.toMatchObject({ diff --git a/server/src/__tests__/idle-sleep-safety.test.ts b/server/src/__tests__/idle-sleep-safety.test.ts index a05af7c7a96..3d4bb27077c 100644 --- a/server/src/__tests__/idle-sleep-safety.test.ts +++ b/server/src/__tests__/idle-sleep-safety.test.ts @@ -5,7 +5,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest import { agentInstructionWorkingCopies, agents, agentApiKeys, agentWakeupRequests, budgetPolicies, companies, companySecretProposals, createDb, adapterAuthSessions, environments, environmentLeases, executionWorkspaces, - feedbackExports, feedbackVotes, + feedbackExports, feedbackVotes, companySkills, companySkillVersions, companySkillRepositorySnapshots, heartbeatRuns, issues, issueWatchdogs, issueThreadInteractions, issueQuestionResponseDeliveries, projects, projectWorkspaces, routines, routineTriggers, toolApplications, toolConnections, workspaceOperations, executionWorkspaceRuntimeLeases, type Db, } from "@paperclipai/db"; @@ -187,6 +187,23 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp return { companyId, agentId, runId }; } + it.each(["true", "false"])("lets optional conversion retries sleep with package storage %s", async enabled => { + const previous = process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; + process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = enabled; + try { + const { companyId } = await seed(); + const [skill] = await db.insert(companySkills).values({ companyId, key: "legacy", slug: "legacy", name: "Legacy", markdown: "original" }).returning(); + await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 1, + fileInventory: [{ path: "SKILL.md", kind: "skill", content: "original" }], migrationAt: new Date(Date.now() + 60 * 60_000) }); + await db.insert(companySkillRepositorySnapshots).values({ companyId, contentHash: "a".repeat(64), + fileInventory: [{ path: "SKILL.md", kind: "skill", content: "original" }], migrationAt: new Date(0) }); + expect(await read()).toEqual(none); + } finally { + if (previous === undefined) delete process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; + else process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = previous; + } + }); + it("retains committed budget stop intent until its exact generation is delivered", async () => { const f = await seed(); const [policy] = await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId, diff --git a/server/src/__tests__/project-skill-sources.test.ts b/server/src/__tests__/project-skill-sources.test.ts index 20838033e00..ed8e6c4f4d9 100644 --- a/server/src/__tests__/project-skill-sources.test.ts +++ b/server/src/__tests__/project-skill-sources.test.ts @@ -8,7 +8,7 @@ import os from 'node:os'; import path from 'node:path'; import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; import { and, eq, sql } from 'drizzle-orm'; -import { activityLog, agents, authUsers, companies, companyMemberships, companySkillSources, companySkillImportBudgets, companySkills, createDb, projects, +import { activityLog, agents, authUsers, companies, companyMemberships, companySkillSources, companySkillImportBudgets, companySkills, companySkillVersions, createDb, projects, projectWorkspaces, projectSkillSources, projectAccessMembers } from '@paperclipai/db'; import { startEmbeddedPostgresTestDatabase, getEmbeddedPostgresTestSupport } from './helpers/embedded-postgres.js'; import { githubFixture } from './helpers/github-skills.js'; @@ -16,6 +16,7 @@ import { enqueueProjectSkillSource, detachProjectSkillSource, projectSkillSetupI import { adoptProjectSkillRepository, projectSkillRepositories, skillSourceSyncService } from '../services/skill-source-sync.js'; import { skillSourceService, type SkillSourceContext } from '../services/skill-sources.js'; import { companySkillService } from '../services/company-skills.js'; +import { skillPackagePaths } from '../services/skill-package-store.js'; import { projectService } from '../services/projects.js'; import { skillAccessService, withSkillActor } from '../services/skill-access.js'; import type { AuthorizationActor } from '../services/authorization.js'; @@ -446,6 +447,7 @@ describe.skipIf(!support.supported)('automatic project skill sources', () => { const source = await skillSourceService(db).detail(companyId, binding!.sourceId!); const skillId = source.entries[0]!.skillId!; let privateSnapshotPath = ''; + let privatePackagePath = ''; const privateDerivatives: string[] = []; for (const user of ['alice', 'bob']) { await db.insert(authUsers).values({ id: user, name: user, email: `${user}@example.test`, createdAt: new Date(), updatedAt: new Date() }); @@ -459,6 +461,16 @@ describe.skipIf(!support.supported)('automatic project skill sources', () => { const [runtime] = await companySkillService(db).listRuntimeSkillEntries(companyId, { onlyKeys: new Set([skill.key]) }); privateSnapshotPath = runtime!.source; await expect(companySkillService(db).importFromSource(companyId, runtime!.source)).rejects.toThrow(/Project-derived skill/); + const [version] = await db.select({ packageId: companySkillVersions.packageSnapshotId }) + .from(companySkillVersions).where(eq(companySkillVersions.id, skill.currentVersionId!)); + expect(version?.packageId).toBeTruthy(); + privatePackagePath = skillPackagePaths(companyId, version!.packageId!).directory; + expect((await fs.stat(privatePackagePath)).isDirectory()).toBe(true); + for (const candidate of [privatePackagePath, path.join(privatePackagePath, 'private'), path.dirname(privatePackagePath)]) { + await expect(companySkillService(db).importFromSource(companyId, candidate)).rejects.toMatchObject({ + status: 422, details: { code: 'skill_package_cache_import_denied' }, + }); + } const fork = await companySkillService(db).forkSkill(companyId, skillId, { slug: 'private-copy' }); expect(fork.skill.accessSourceId).not.toBe(source.id); expect(fork.skill.accessSourceId).toBeTruthy(); @@ -475,6 +487,9 @@ describe.skipIf(!support.supported)('automatic project skill sources', () => { await expect(skillSourceService(db).branches(companyId, source.id, {}, { ...context, read: () => branchRead })).rejects.toThrow(/not found/); expect(branchRead).not.toHaveBeenCalled(); await expect(companySkillService(db).importFromSource(companyId, privateSnapshotPath)).rejects.toThrow(/not found/); + await expect(companySkillService(db).importFromSource(companyId, privatePackagePath)).rejects.toMatchObject({ + status: 422, details: { code: 'skill_package_cache_import_denied' }, + }); expect(await companySkillService(db).getById(companyId, skillId)).toBeNull(); expect(await companySkillService(db).readFile(companyId, skillId, 'SKILL.md')).toBeNull(); expect((await skillSourceService(db).list(companyId)).some(item => item.id === source.id)).toBe(false); diff --git a/server/src/__tests__/skill-package-store.test.ts b/server/src/__tests__/skill-package-store.test.ts new file mode 100644 index 00000000000..8b9d824e734 --- /dev/null +++ b/server/src/__tests__/skill-package-store.test.ts @@ -0,0 +1,986 @@ +import { skillPackageWork } from "../services/skill-package-work.js"; +import * as packageCache from "../services/skill-package-cache.js"; +import * as recovery from "../services/skill-package-recovery.js"; +import type { GitHubRead } from "../services/github-skill-source.js"; +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { companies, companySkills, companySkillVersions, companySkillRepositorySnapshots as snapshots, + skillPackageDeletions, createDb, withDedicatedDbConnection } from "@paperclipai/db"; +import * as dbModule from "@paperclipai/db"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import * as storageModule from "../storage/index.js"; +import { createStorageService } from "../storage/service.js"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { companySkillService } from "../services/company-skills.js"; +import { skillPackageStore, skillPackagePaths, withSkillPackageReadScope } from "../services/skill-package-store.js"; +import { skillPackageMaintenance } from "../services/skill-package-maintenance.js"; +import { skillPackageManifest, validateSkillPackageManifest, writeSkillPackageArchive, extractSkillPackageArchive, + verifySkillPackageDirectory } from "../services/skill-package-files.js"; +import { skillFileBytes, skillSnapshotHash } from "../services/skill-snapshot.js"; +import { readCustomerSuccessResource } from "../services/customer-success-inspection.js"; +import { repositorySkillProjection } from "../services/skill-repository-package.js"; +import { hasRestorableSnapshot } from "../services/skill-version-cache.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("external skill packages", () => { + let database: Awaited>; + let db: ReturnType; + let home: string; + let storage: storageModule.StorageService; + const previousHome = process.env.PAPERCLIP_HOME; + const files: CompanySkillVersionFileInventoryEntry[] = [ + { path: "skills/one/SKILL.md", kind: "skill", content: "---\nname: one\ndescription: Test one\n---\nUse ../../shared/run.sh\n" }, + { path: "skills/two/SKILL.md", kind: "skill", content: "---\nname: two\ndescription: Test two\n---\nUse ../../shared/run.sh\n" }, + { path: "shared/run.sh", kind: "script", content: "#!/bin/sh\necho original\n", executable: true }, + { path: "assets/binary", kind: "reference", content: Buffer.from([0, 255, 7]).toString("base64"), encoding: "base64" }, + { path: "assets/base64-text", kind: "reference", content: Buffer.from("text encoded explicitly").toString("base64"), encoding: "base64" }, + ]; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("skill-packages-"); + db = createDb(database.connectionString, { maxConnections: 1 }); + home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-packages-")); + process.env.PAPERCLIP_HOME = home; + storage = createStorageService(createLocalDiskStorageProvider(path.join(home, "objects"))); + }); + afterEach(() => vi.restoreAllMocks()); + afterAll(async () => { + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + await db?.$client.end({ timeout: 1 }); + await database?.cleanup(); + if (home) await fs.rm(home, { recursive: true, force: true }); + }); + async function company() { + const id = randomUUID(); + await db.insert(companies).values({ id, name: "Package test", issuePrefix: `P${id.slice(0, 6)}` }); + return id; + } + async function fixture(recoverable = false) { + const companyId = await company(); + vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage); + const store = skillPackageStore(db, storage); + const recipe = recoverable ? { version: 1 as const, sourceId: randomUUID(), repositoryId: "123", repositoryUrl: "https://github.com/acme/skills", + commitSha: "a".repeat(40), skillPath: null, manifest: skillPackageManifest(files) } : undefined; + const id = await store.prepare(companyId, files, false, undefined, recipe); + const skills = []; + for (const name of ["one", "two"]) { + const [skill] = await db.insert(companySkills).values({ companyId, key: `company/${companyId}/${name}`, slug: name, + name, markdown: files.find(file => file.path === `skills/${name}/SKILL.md`)!.content, sourceType: "github", + metadata: { skillSourceId: randomUUID(), snapshotHash: "saved" } }).returning(); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 1, + repositorySnapshotId: id, repositorySkillPath: `skills/${name}/SKILL.md`, packageSnapshotId: id, migrationAt: null, + packageFileMap: [{ path: "SKILL.md", kind: "skill", sourcePath: `skills/${name}/SKILL.md` }] }).returning(); + await db.update(companySkills).set({ currentVersionId: version!.id }).where(eq(companySkills.id, skill!.id)); + skills.push({ skill: skill!, version: version! }); + } + const service = companySkillService(db); + const resolve = () => service.listRuntimeSkillEntries(companyId, { onlyKeys: new Set(skills.map(({ skill }) => skill.key)), snapshotLatest: true }); + return { companyId, id, store, skills, service, resolve }; + } + + it("orders skill and package locks consistently with an edit's dedicated upload connection", async () => { + const f = await fixture(); + const skill = f.skills[0]!.skill; + const options = { skill, skipInventoryRefresh: true, fileInventory: files, packageSnapshotId: f.id }; + let waitingOnSkill!: () => void; + const waiting = new Promise(resolve => { waitingOnSkill = resolve; }); + const debug = db.$client.options.debug; + db.$client.options.debug = (_connection, query) => { + if (query.includes('from "company_skills"') && query.includes('for update')) waitingOnSkill(); + }; + let saving: Promise | undefined; + try { + await withDedicatedDbConnection(db, connection => connection.transaction(async tx => { + // updateFile owns this lock before its package is prepared separately. + await tx.select({ id: companySkills.id }).from(companySkills).where(eq(companySkills.id, skill.id)).for("update"); + saving = f.service.createVersion(f.companyId, skill.id, {}, null, options); + await waiting; + await withDedicatedDbConnection(db, async upload => { + // A regression fails promptly and releases the outer lock instead of + // leaving a hidden cross-connection wait cycle hanging the suite. + await upload.execute(sql`set statement_timeout = '3s'`); + expect(await skillPackageStore(upload, storage).prepare(f.companyId, files)).toBe(f.id); + }); + await f.service.createVersion(f.companyId, skill.id, {}, null, { ...options, database: tx }); + })); + await saving; + expect(await f.service.listVersions(f.companyId, skill.id)).toHaveLength(3); + } finally { db.$client.options.debug = debug; await saving; } + }, 15_000); + + it("follows a package published between version metadata and summary reads", async () => { + const companyId = await company(); + const [skill] = await db.insert(companySkills).values({ companyId, key: "summary", slug: "summary", name: "Summary", markdown: files[0]!.content }).returning(); + const [stale] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 1, fileInventory: files }).returning(); + await skillPackageMaintenance(db, storage).migrateVersions({ id: stale!.id, apply: true, clearLegacy: true }); + expect(await skillPackageStore(db, storage).versionInventory(stale!)).toEqual(files.map(({ path, kind }) => ({ path, kind }))); + }); + + it("forks a packaged repository inside an outer transaction with a one-connection pool", async () => { + const f = await fixture(); + const fork = await db.transaction(tx => companySkillService(tx as unknown as typeof db, { packageDb: db }) + .createLocalSkill(f.companyId, { name: "Idempotent fork", slug: "idempotent-fork", forkedFromSkillId: f.skills[0]!.skill.id })); + const saved = await f.service.getVersion(f.companyId, fork.id, fork.currentVersionId!); + expect(saved!.fileInventory).toEqual(expect.arrayContaining([ + expect.objectContaining({ path: ".paperclip-repository/shared/run.sh", content: files[2]!.content }), + ])); + }, 15_000); + + it("lists version metadata without reading archives, while inspection reads the requested historical bytes", async () => { + const f = await fixture(); + const { skill, version } = f.skills[0]!; + await db.update(companySkills).set({ markdown: "New current markdown" }).where(eq(companySkills.id, skill.id)); + const read = vi.spyOn(storage, "getObject"); + const history = await f.service.listVersions(f.companyId, skill.id); + expect(history[0]!.fileInventory).toEqual([{ path: "SKILL.md", kind: "skill" }]); + expect(read.mock.calls.filter(call => call[1].endsWith("files.tar.gz"))).toHaveLength(0); + await db.update(companySkillVersions).set({ packageFileMap: [ + ...version.packageFileMap!, { path: "run.sh", kind: "script", sourcePath: "shared/run.sh" }, + ] }).where(eq(companySkillVersions.id, version.id)); + const script = await readCustomerSuccessResource(db, storage, { + operation: "skills.file", companyId: f.companyId, resourceId: skill.id, versionId: version.id, path: "run.sh", + }) as { content: string }; + expect(Buffer.from(script.content, "base64").toString()).toBe(files[2]!.content); + const result = await readCustomerSuccessResource(db, storage, { + operation: "skills.file", companyId: f.companyId, resourceId: skill.id, versionId: version.id, path: "SKILL.md", + }) as { content: string }; + expect(Buffer.from(result.content, "base64").toString()).toBe(files[0]!.content); + const missing = await readCustomerSuccessResource(db, storage, { + operation: "skills.file", companyId: f.companyId, resourceId: skill.id, versionId: version.id, path: "not-saved.md", + }); + expect(missing).toEqual({ available: false, reason: "skill_file_not_snapshotted" }); + }); + + it("does not block a warm turn behind a paused archive read for retention", async () => { + const f = await fixture(); + await f.resolve(); + await fs.rm(skillPackagePaths(f.companyId, f.id).directory, { recursive: true, force: true }); + let release!: () => void; + let entered!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + const reading = new Promise(resolve => { entered = resolve; }); + const original = storage.getObject.bind(storage); + vi.spyOn(storage, "getObject").mockImplementation(async (companyId, key, options) => { + if (key.endsWith("files.tar.gz")) { entered(); await blocked; } + return original(companyId, key, options); + }); + const cleanup = hasRestorableSnapshot(db, f.companyId, f.skills[0]!.version.id); + try { + await reading; + const warm = await f.resolve(); + expect(warm.map(entry => entry.sourceStatus)).toEqual(["available", "available"]); + } finally { release(); await cleanup; } + expect(await fs.stat(skillPackagePaths(f.companyId, f.id).directory).catch(() => null)).toBeNull(); + }, 15_000); + + it("skips an in-flight publication without waiting on its row or the warm preparation lock", async () => { + const f = await fixture(true); + await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.companyId, f.companyId)); + await db.update(snapshots).set({ createdAt: new Date(0), maintenanceAt: new Date(0) }).where(eq(snapshots.id, f.id)); + let retirementStarted!: () => void; + const waiting = new Promise(resolve => { retirementStarted = resolve; }); + const debug = db.$client.options.debug; + db.$client.options.debug = (_connection, query) => { + if (query.includes('company_skill_repository_snapshots') && query.includes('for update')) retirementStarted(); + }; + let sweep: Promise | undefined; + try { + await withDedicatedDbConnection(db, connection => connection.transaction(async tx => { + await tx.select({ id: snapshots.id }).from(snapshots).where(eq(snapshots.id, f.id)).for("update"); + await tx.update(companySkills).set({ currentVersionId: f.skills[0]!.version.id }).where(eq(companySkills.id, f.skills[0]!.skill.id)); + sweep = skillPackageWork(db, storage).tick(new AbortController().signal); + await waiting; + await sweep; + })); + await sweep; + expect((await f.store.descriptor(f.companyId, f.id)).storage).not.toBeNull(); + } finally { db.$client.options.debug = debug; await sweep; } + }, 15_000); + + it("refuses to publish a package retired before the publication row lock", async () => { + const f = await fixture(true); + await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.companyId, f.companyId)); + await db.update(snapshots).set({ storage: null }).where(eq(snapshots.id, f.id)); + await expect(f.service.createVersion(f.companyId, f.skills[0]!.skill.id, {}, null, { + skill: f.skills[0]!.skill, skipInventoryRefresh: true, fileInventory: files, + packageSnapshotId: f.id, repositorySnapshotId: f.id, repositorySkillPath: files[0]!.path, + })).rejects.toThrow(/changed during publication/); + const [skill] = await db.select().from(companySkills).where(eq(companySkills.id, f.skills[0]!.skill.id)); + expect(skill!.currentVersionId).toBeNull(); + }); + + it("converts a legacy fork with the scanner's deepest accepted repository path", async () => { + const companyId = await company(); + const deep = [...Array(63).fill("d"), "SKILL.md"].join("/"); + const projected = repositorySkillProjection([{ ...files[0]!, path: deep }], deep); + const [skill] = await db.insert(companySkills).values({ companyId, key: "deep", slug: "deep", name: "Deep", markdown: projected[0]!.content }).returning(); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 1, fileInventory: projected }).returning(); + await skillPackageMaintenance(db, storage).migrateVersions({ id: version!.id, apply: true, clearLegacy: true }); + const [saved] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version!.id)); + expect(saved!.migrationAt).toBeNull(); + expect(await skillPackageStore(db, storage).versionFiles(saved!)).toEqual(projected); + }); + + it("automatically converts legacy rows, clears verified payloads and preserves version identities", async () => { + const companyId = await company(); + const [snapshot] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const [skill] = await db.insert(companySkills).values({ companyId, key: "legacy", slug: "legacy", name: "Legacy", markdown: files[0]!.content }).returning(); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 7, + repositorySnapshotId: snapshot!.id, repositorySkillPath: files[0]!.path, + fileInventory: [{ ...files[0]!, path: "SKILL.md" }] }).returning(); + const work = skillPackageWork(db, storage); + await work.tick(new AbortController().signal); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.id, snapshot!.id)); + const [converted] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version!.id)); + expect(saved!.fileInventory).toEqual([]); + expect(saved!.migrationAt).toBeNull(); + expect(saved!.storage).not.toBeNull(); + expect(converted!.fileInventory).toEqual([]); + expect(converted!.migrationAt).toBeNull(); + expect(converted!.packageSnapshotId).toBe(snapshot!.id); + expect(converted!.revisionNumber).toBe(7); + expect(await skillPackageStore(db, storage).versionFiles(converted!)).toEqual([{ ...files[0]!, path: "SKILL.md" }]); + }); + + it.each(["file", "folder"] as const)("restores a deleted %s when the replacement package upload fails", async target => { + const companyId = await company(); + vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage); + const service = companySkillService(db); + const skill = await service.createLocalSkill(companyId, { name: "Delete rollback", slug: "delete-rollback", markdown: "# Keep the skill\n" }); + const binary = Buffer.from([0, 255, 7, 128]); + await service.updateFile(companyId, skill.id, "bundle/probe.sh", "#!/bin/sh\nprintf kept\n", null, { executable: true }); + await service.updateFile(companyId, skill.id, "bundle/nested/data.bin", binary.toString("base64"), null, { encoding: "base64" }); + await service.updateFile(companyId, skill.id, "keep.txt", "Unrelated file\n"); + const root = skill.sourceLocator!; + await fs.chmod(path.join(root, "bundle/probe.sh"), 0o751); + await fs.mkdir(path.join(root, "bundle/empty")); + await fs.mkdir(path.join(root, "bundle/.git")); + await fs.writeFile(path.join(root, "bundle/.git/local-state"), "Unlisted bytes\n"); + const [before] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id)); + const versionsBefore = await db.select({ id: companySkillVersions.id }).from(companySkillVersions) + .where(eq(companySkillVersions.companySkillId, skill.id)); + const input = { path: target === "file" ? "bundle/probe.sh" : "bundle", target }; + const upload = vi.spyOn(storage, "putFile").mockRejectedValue(new Error("Object storage unavailable")); + try { + await expect(service.deleteFile(companyId, skill.id, input)).rejects.toThrow("Object storage unavailable"); + expect(upload).toHaveBeenCalled(); + } finally { upload.mockRestore(); } + expect(await fs.readFile(path.join(root, "bundle/probe.sh"), "utf8")).toBe("#!/bin/sh\nprintf kept\n"); + expect((await fs.stat(path.join(root, "bundle/probe.sh"))).mode & 0o777).toBe(0o751); + expect(await fs.readFile(path.join(root, "bundle/nested/data.bin"))).toEqual(binary); + expect((await fs.stat(path.join(root, "bundle/empty"))).isDirectory()).toBe(true); + expect(await fs.readFile(path.join(root, "bundle/.git/local-state"), "utf8")).toBe("Unlisted bytes\n"); + const [after] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id)); + expect(after).toEqual(before); + expect(await db.select({ id: companySkillVersions.id }).from(companySkillVersions) + .where(eq(companySkillVersions.companySkillId, skill.id))).toEqual(versionsBefore); + // Failed uploads keep a durable claim for reconciliation. Simulate its + // expiry before retrying the same package with recovered storage. + await db.update(snapshots).set({ upload: sql`jsonb_set(${snapshots.upload}, '{expiresAt}', to_jsonb(${new Date(0).toISOString()}::text))` }) + .where(and(eq(snapshots.companyId, companyId), sql`${snapshots.upload} is not null`)); + const deleted = await service.deleteFile(companyId, skill.id, input); + expect(deleted.deletedPaths).toContain("bundle/probe.sh"); + await expect(fs.stat(path.join(root, input.path))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await fs.readFile(path.join(root, "keep.txt"), "utf8")).toBe("Unrelated file\n"); + expect((await service.getVersion(companyId, skill.id, before!.currentVersionId!))!.fileInventory) + .toEqual(expect.arrayContaining([expect.objectContaining({ path: "bundle/probe.sh", executable: true })])); + }); + + it.each(["committed", "rolled_back", "newer_edit", "unavailable"] as const)("reconciles a lost deletion commit reply when %s", async outcome => { + const companyId = await company(); + vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage); + const service = companySkillService(db); + const skill = await service.createLocalSkill(companyId, { name: "Commit reply", slug: "commit-reply", markdown: "# Keep\n" }); + await service.updateFile(companyId, skill.id, "notes.txt", "Original notes\n"); + await service.updateFile(companyId, skill.id, "keep.txt", "Keep\n"); + const before = (await service.getById(companyId, skill.id))!; + let deletionFinished = false; + let backupDirectory: string | undefined; + const mkdtemp = fs.mkdtemp.bind(fs); + vi.spyOn(fs, "mkdtemp").mockImplementation(async (...args) => { + const directory = await mkdtemp(...args); + if (String(args[0]).includes("paperclip-skill-delete-")) backupDirectory = directory as string; + return directory; + }); + const ambiguousDb = new Proxy(db, { + get(target, property, receiver) { + if (property === "transaction") return async (callback: (tx: any) => Promise, ...options: any[]) => { + if (deletionFinished && outcome === "unavailable") throw new Error("Database unavailable"); + const result = await target.transaction(async tx => { + const value = await callback(tx); + if (value?.deletedPaths) { + deletionFinished = true; + if (outcome === "rolled_back") throw new Error("Lost commit reply"); + } + return value; + }, ...options); + if (!result?.deletedPaths) return result; + if (outcome === "newer_edit") await service.updateFile(companyId, skill.id, "notes.txt", "Newer edit\n"); + throw new Error("Lost commit reply"); + }; + return Reflect.get(target, property, receiver); + }, + }); + try { + await expect(companySkillService(ambiguousDb, { packageDb: db }).deleteFile(companyId, skill.id, + { path: "notes.txt", target: "file" })).rejects.toThrow("Lost commit reply"); + expect(deletionFinished).toBe(true); + const [saved] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id)); + if (outcome === "rolled_back") { + expect(saved!.currentVersionId).toBe(before.currentVersionId); + expect(await fs.readFile(path.join(skill.sourceLocator!, "notes.txt"), "utf8")).toBe("Original notes\n"); + } else if (outcome === "newer_edit") { + expect(await fs.readFile(path.join(skill.sourceLocator!, "notes.txt"), "utf8")).toBe("Newer edit\n"); + } else { + expect(saved!.currentVersionId).not.toBe(before.currentVersionId); + await expect(fs.stat(path.join(skill.sourceLocator!, "notes.txt"))).rejects.toMatchObject({ code: "ENOENT" }); + } + if (outcome === "unavailable" || outcome === "newer_edit") { + expect(await fs.readFile(path.join(backupDirectory!, "source"), "utf8")).toBe("Original notes\n"); + } else await expect(fs.stat(backupDirectory!)).rejects.toMatchObject({ code: "ENOENT" }); + } finally { + if (backupDirectory) await fs.rm(backupDirectory, { recursive: true, force: true }); + } + }); + + it("expires an unused Git package and reacquires its exact bytes without a migration command", async () => { + const f = await fixture(true); + const recipe = { version: 1 as const, sourceId: randomUUID(), repositoryId: "123", repositoryUrl: "https://github.com/acme/skills", + commitSha: "a".repeat(40), skillPath: null, manifest: skillPackageManifest(files) }; + await f.store.prepare(f.companyId, files, true, undefined, recipe); + await f.store.files(f.companyId, f.id); + await db.update(snapshots).set({ createdAt: new Date(Date.now() - 40 * 86_400_000), maintenanceAt: new Date(0), migrationAt: null }).where(eq(snapshots.id, f.id)); + // Current versions stay pinned even after their cache age expires. + await skillPackageWork(db, storage).tick(new AbortController().signal); + expect((await f.store.descriptor(f.companyId, f.id)).storage).not.toBeNull(); + for (const { skill } of f.skills) await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.id, skill.id)); + await db.update(snapshots).set({ maintenanceAt: new Date(0) }).where(eq(snapshots.id, f.id)); + await skillPackageWork(db, storage).tick(new AbortController().signal); + expect((await f.store.descriptor(f.companyId, f.id)).storage).toBeNull(); + expect(await fs.stat(skillPackagePaths(f.companyId, f.id).directory).catch(() => null)).toBeNull(); + const entries = recipe.manifest.files.map((file, i) => ({ path: file.path, mode: file.executable ? "100755" : "100644", type: "blob", size: file.size, sha: String(i).repeat(40) })); + const read = Object.assign(vi.fn(async () => ({ id: 123 })), { openSnapshot: vi.fn(async () => ({ commitSha: recipe.commitSha, entries, + readBlob: async (entry: typeof entries[number]) => skillFileBytes(files[entries.indexOf(entry)]!), release: async () => {} })) }) as unknown as GitHubRead; + const authorization = vi.spyOn(recovery, "skillPackageRecoveryReader").mockResolvedValue(read); + expect(await f.store.files(f.companyId, f.id)).toEqual(files); + expect(read.openSnapshot).toHaveBeenCalledOnce(); + await f.store.files(f.companyId, f.id); + expect(authorization).toHaveBeenCalledOnce(); + await fs.rm(skillPackagePaths(f.companyId, f.id).directory, { recursive: true }); + authorization.mockRejectedValue(new Error("grant revoked")); + await expect(f.store.files(f.companyId, f.id)).rejects.toThrow("grant revoked"); + await expect(f.store.files(f.companyId, f.id)).rejects.toThrow("backoff"); + expect(authorization).toHaveBeenCalledTimes(2); + }); + + it("recovers missing archive and manifest objects while a published receipt still exists", async () => { + const companyId = await company(); + const store = skillPackageStore(db, storage); + const recipe = { version: 1 as const, sourceId: randomUUID(), repositoryId: "123", repositoryUrl: "https://github.com/acme/skills", + commitSha: "a".repeat(40), skillPath: null, manifest: skillPackageManifest(files) }; + const id = await store.prepare(companyId, files, true, undefined, recipe); + const receipt = (await store.descriptor(companyId, id)).storage!; + await storage.deleteObject(companyId, receipt.archiveKey); + await storage.deleteObject(companyId, receipt.manifestKey); + await expect(recovery.skillPackageRecoveryReader(db, companyId, recipe)).rejects.toThrow("Reconnect its original repository"); + const entries = recipe.manifest.files.map((file, i) => ({ path: file.path, mode: file.executable ? "100755" : "100644", type: "blob", size: file.size, sha: String(i).repeat(40) })); + const read = Object.assign(vi.fn(async () => ({ id: 123 })), { openSnapshot: vi.fn(async () => ({ commitSha: recipe.commitSha, entries, + readBlob: async (entry: typeof entries[number]) => skillFileBytes(files[entries.indexOf(entry)]!), release: async () => {} })) }) as unknown as GitHubRead; + vi.spyOn(recovery, "skillPackageRecoveryReader").mockResolvedValue(read); + // PostgreSQL JSONB has reordered recipe object keys; its canonical bytes must + // still agree with the original externally written manifest checksum. + expect(await store.files(companyId, id)).toEqual(files); + expect(read.openSnapshot).toHaveBeenCalledOnce(); + }); + + it("retains referenced historical bytes without an exact Git recipe", async () => { + const f = await fixture(); + for (const { skill } of f.skills) await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.id, skill.id)); + await db.update(snapshots).set({ createdAt: new Date(0), migrationAt: null, maintenanceAt: new Date(0) }).where(eq(snapshots.id, f.id)); + await skillPackageWork(db, storage).tick(new AbortController().signal); + expect((await f.store.descriptor(f.companyId, f.id)).storage).not.toBeNull(); + expect(await f.store.files(f.companyId, f.id)).toEqual(files); + }); + + it("retains authored history when a Git import shares its package and later loses access", async () => { + const f = await fixture(); + const [authored] = await db.insert(companySkillVersions).values({ companyId: f.companyId, companySkillId: f.skills[0]!.skill.id, + revisionNumber: 2, packageSnapshotId: f.id, migrationAt: null }).returning(); + const recipe = { version: 1 as const, sourceId: randomUUID(), repositoryId: "123", repositoryUrl: "https://github.com/acme/skills", + commitSha: "a".repeat(40), skillPath: null, manifest: skillPackageManifest(files) }; + expect(await f.store.prepare(f.companyId, files, true, undefined, recipe)).toBe(f.id); + for (const { skill } of f.skills) await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.id, skill.id)); + await db.update(snapshots).set({ createdAt: new Date(0), migrationAt: null, maintenanceAt: new Date(0) }).where(eq(snapshots.id, f.id)); + await skillPackageWork(db, storage).tick(new AbortController().signal); + expect((await f.store.descriptor(f.companyId, f.id)).storage).not.toBeNull(); + await fs.rm(skillPackagePaths(f.companyId, f.id).directory, { recursive: true, force: true }); + const upstream = vi.spyOn(recovery, "skillPackageRecoveryReader").mockRejectedValue(new Error("grant revoked")); + expect(await f.store.versionFiles(authored!)).toEqual(files); + expect(upstream).not.toHaveBeenCalled(); + }); + + it("keeps authored durability when it joins an in-flight Git package upload", async () => { + const companyId = await company(); + const store = skillPackageStore(db, storage); + const recipe = { version: 1 as const, sourceId: randomUUID(), repositoryId: "123", repositoryUrl: "https://github.com/acme/skills", + commitSha: "a".repeat(40), skillPath: null, manifest: skillPackageManifest(files) }; + let release!: () => void; + let entered!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + const uploading = new Promise(resolve => { entered = resolve; }); + const put = storage.putFile.bind(storage); + vi.spyOn(storage, "putFile").mockImplementationOnce(async input => { entered(); await blocked; return put(input); }); + const git = store.prepare(companyId, files, true, undefined, recipe); + await uploading; + const authored = store.prepare(companyId, files); + release(); + const ids = await Promise.all([git, authored]); + expect(ids[0]).toBe(ids[1]); + expect((await store.descriptor(companyId, ids[0]!)).storage?.durable).toBe(true); + expect(await db.select().from(skillPackageDeletions).where(eq(skillPackageDeletions.companyId, companyId))).toEqual([]); + }); + + it("preserves version kinds and encodings when identical package bytes are deduplicated", async () => { + const companyId = await company(); + const original: CompanySkillVersionFileInventoryEntry[] = [{ path: "SKILL.md", kind: "skill", content: files[0]!.content }, + { path: "helper", kind: "other", content: "#!/bin/sh\necho same\n" }]; + const changed: CompanySkillVersionFileInventoryEntry[] = [original[0]!, { ...original[1]!, kind: "script", encoding: "base64", + content: Buffer.from(original[1]!.content).toString("base64") }]; + const store = skillPackageStore(db, storage); + const packageId = await store.prepare(companyId, original); + vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage); + const [skill] = await db.insert(companySkills).values({ companyId, key: "metadata", slug: "metadata", name: "Metadata", markdown: original[0]!.content }).returning(); + const service = companySkillService(db); + const saved = await service.createVersion(companyId, skill!.id, {}, null, { skill: skill!, skipInventoryRefresh: true, fileInventory: changed }); + expect((await service.getVersion(companyId, skill!.id, saved.id))!.fileInventory).toEqual(changed); + const [legacy] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 2, fileInventory: changed }).returning(); + await skillPackageMaintenance(db, storage).migrateVersions({ id: legacy!.id, apply: true, clearLegacy: true }); + const [converted] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, legacy!.id)); + expect(converted!.packageSnapshotId).toBe(packageId); + expect(converted!.fileInventory).toEqual([]); + expect(await store.versionFiles(converted!)).toEqual(changed); + }); + + it("does not delete an orphan whose expired upload was reclaimed after selection", async () => { + const companyId = await company(); + const upload = { token: randomUUID(), provider: "local_disk" as const, archiveKey: `${companyId}/old/archive`, manifestKey: `${companyId}/old/manifest`, expiresAt: new Date(0).toISOString() }; + const [row] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), createdAt: new Date(0), upload }).returning(); + const renewed = { ...upload, token: randomUUID(), expiresAt: new Date(Date.now() + 600_000).toISOString() }; + const racingDb = new Proxy(db, { get(target, key) { + if (key !== "delete") return Reflect.get(target, key); + return (table: typeof snapshots) => ({ where: (predicate: Parameters["where"]>[0]) => ({ returning: async () => { + await db.update(snapshots).set({ upload: renewed }).where(eq(snapshots.id, row!.id)); + return db.delete(table).where(predicate).returning({ id: snapshots.id }); + } }) }); + } }); + expect((await skillPackageMaintenance(racingDb, storage).collectOrphans({ companyId, apply: true })).removed).toBe(0); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.id, row!.id)); + expect(saved!.upload).toEqual(renewed); + }); + + it("restores one shared archive, then prepares a warm turn without object reads or SQL file bodies", async () => { + const f = await fixture(); + const read = vi.spyOn(storage, "getObject"); + const cold = await f.resolve(); + expect(cold.map(entry => entry.sourceStatus)).toEqual(["available", "available"]); + expect(read.mock.calls.filter(call => call[1].endsWith("files.tar.gz"))).toHaveLength(1); + expect(read.mock.calls.filter(call => call[1].endsWith("manifest.json"))).toHaveLength(1); + const [row] = await db.select().from(snapshots).where(eq(snapshots.id, f.id)); + expect(row!.fileInventory).toEqual([]); + const sql: string[] = []; + const oldDebug = db.$client.options.debug; + db.$client.options.debug = (_connection, query) => { sql.push(query); }; + read.mockClear(); + try { expect(await f.resolve()).toEqual(cold); } + finally { db.$client.options.debug = oldDebug; } + expect(read).not.toHaveBeenCalled(); + expect(sql.length).toBeLessThanOrEqual(6); + for (const query of sql.filter(query => /from "company_skill_(repository_snapshots|versions)"/.test(query))) { + // Bundled release validation returns only a DB-side digest. + if (!query.includes("encode(sha256(")) expect(query).not.toMatch(/"file_inventory"/); + } + expect((await f.service.getVersion(f.companyId, f.skills[0]!.skill.id, f.skills[0]!.version.id))!.fileInventory) + .toEqual([{ path: "SKILL.md", kind: "skill", content: files[0]!.content }]); + const restored = await f.store.files(f.companyId, f.id); + expect(restored.map(skillFileBytes)).toEqual(files.map(skillFileBytes)); + await expect(f.store.files(await company(), f.id)).rejects.toThrow("unavailable"); + }); + + it("repairs a same-size cached edit and restores old versions without upstream access", async () => { + const f = await fixture(); + const entries = await f.resolve(); + const target = path.join(entries[0]!.source, ".paperclip-repository/shared/run.sh"); + const stat = await fs.stat(target); + await fs.chmod(target, 0o755); + await fs.writeFile(target, "#!/bin/sh\necho modified\n"); + await fs.utimes(target, stat.atime, stat.mtime); + const read = vi.spyOn(storage, "getObject"); + await f.resolve(); + expect(await fs.readFile(target, "utf8")).toBe(files[2]!.content); + expect(read).not.toHaveBeenCalled(); + const upstream = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("offline")); + for (const entry of entries) await fs.rm(entry.source, { recursive: true }); + await fs.rm(skillPackagePaths(f.companyId, f.id).directory, { recursive: true }); + expect((await f.resolve()).every(entry => entry.sourceStatus === "available")).toBe(true); + expect(upstream).not.toHaveBeenCalled(); + }); + + it("refuses cache eviction when durable objects are missing, even after an earlier verification", async () => { + const f = await fixture(); + await f.resolve(); + expect(await hasRestorableSnapshot(db, f.companyId, f.skills[0]!.version.id)).toBe(true); + const receipt = (await f.store.descriptor(f.companyId, f.id)).storage!; + await storage.deleteObject(f.companyId, receipt.archiveKey); + await expect(hasRestorableSnapshot(db, f.companyId, f.skills[0]!.version.id)).rejects.toThrow(); + // Existing local verified files remain usable while storage is unavailable. + expect((await f.resolve()).every(entry => entry.sourceStatus === "available")).toBe(true); + }); + + it("reads verified cached versions offline while independently checking durable bytes and rejecting corrupt files", async () => { + const f = await fixture(); + await f.store.files(f.companyId, f.id); + const read = vi.spyOn(storage, "getObject").mockRejectedValue(new Error("storage offline")); + const saved = f.skills[0]!; + expect((await f.service.getVersion(f.companyId, saved.skill.id, saved.version.id))!.fileInventory) + .toEqual([{ path: "SKILL.md", kind: "skill", content: files[0]!.content }]); + expect(await f.store.versionFiles({ ...saved.version, packageFileMap: null })).toEqual(files); + expect(read).not.toHaveBeenCalled(); + await expect(f.store.verifyDurable(f.companyId, f.id)).rejects.toThrow("storage offline"); + const target = path.join(skillPackagePaths(f.companyId, f.id).directory, files[0]!.path); + await fs.chmod(target, 0o600); + await fs.writeFile(target, "x".repeat(Buffer.byteLength(files[0]!.content))); + await expect(f.store.versionFiles(saved.version)).rejects.toThrow("storage offline"); + }); + + it("coalesces cold version reads and retains the verified package for subsequent reads", async () => { + const f = await fixture(); + await f.resolve(); + const root = skillPackagePaths(f.companyId, f.id).directory; + await fs.rm(root, { recursive: true }); + const read = vi.spyOn(storage, "getObject"); + const saved = f.skills[0]!; + const get = () => f.service.getVersion(f.companyId, saved.skill.id, saved.version.id); + const [first, second, third] = await Promise.all([get(), get(), get()]); + expect(first!.fileInventory).toEqual([{ path: "SKILL.md", kind: "skill", content: files[0]!.content }]); + expect(second).toEqual(first); + expect(third).toEqual(first); + expect(await get()).toEqual(first); + expect(await f.store.readVersionFile(saved.version, "SKILL.md")).toMatchObject({ content: files[0]!.content }); + expect(read.mock.calls.filter(call => call[1].endsWith("files.tar.gz"))).toHaveLength(1); + expect(await verifySkillPackageDirectory(root, skillPackageManifest(files).files)).toBe(true); + }); + + it("shares a cold manifest while allowing one caller to cancel without aborting another", async () => { + const f = await fixture(); + const get = storage.getObject.bind(storage); + let entered!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + let storageSignal: AbortSignal | undefined; + const read = vi.spyOn(storage, "getObject").mockImplementation(async (...args) => { + storageSignal = args[2]?.signal; + entered(); + await blocked; + return get(...args); + }); + const cancellation = new AbortController(); + const first = f.store.manifest(f.companyId, f.id, cancellation.signal).catch(error => error); + await started; + const second = f.store.manifest(f.companyId, f.id); + try { + // The shared one-slot pool places this query after the second descriptor. + await db.select({ id: snapshots.id }).from(snapshots).where(eq(snapshots.id, f.id)); + cancellation.abort(new Error("Caller stopped")); + expect((await first).message).toBe("Caller stopped"); + expect(storageSignal?.aborted).toBe(false); + release(); + expect(await second).toEqual(skillPackageManifest(files)); + expect(read).toHaveBeenCalledOnce(); + } finally { release(); await Promise.allSettled([first, second]); } + }); + + it("keeps warm preparation responsive while a read-triggered cache trim verifies storage", async () => { + const f = await fixture(); + await f.resolve(); + const paths = skillPackagePaths(f.companyId, f.id); + // Exceed the directory bound without manufacturing large file payloads. + for (let index = 0; index < 63; index++) { + const orphan = path.join(paths.parent, randomUUID()); + await fs.mkdir(orphan); + const later = new Date(Date.now() + 60_000); + await fs.utimes(orphan, later, later); + } + const changed = files.map((file, index) => index ? file : { ...file, content: `${file.content}Updated\n` }); + const id = await f.store.prepare(f.companyId, changed); + const [version] = await db.insert(companySkillVersions).values({ companyId: f.companyId, + companySkillId: f.skills[0]!.skill.id, revisionNumber: 2, packageSnapshotId: id, migrationAt: null, + packageFileMap: f.skills[0]!.version.packageFileMap }).returning(); + const receipt = (await f.store.descriptor(f.companyId, f.id)).storage!; + const get = storage.getObject.bind(storage); + let entered!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + vi.spyOn(storage, "getObject").mockImplementation(async (...args) => { + if (args[1] === receipt.manifestKey) { entered(); await blocked; } + return get(...args); + }); + const reading = f.store.versionFiles(version!); + let warming: ReturnType | undefined; + let timeout: ReturnType | undefined; + try { + await started; + warming = f.resolve(); + expect((await Promise.race([warming, new Promise((_resolve, reject) => { + timeout = setTimeout(() => reject(new Error("Warm preparation waited for eviction storage")), 1_000); + })])).every(entry => entry.sourceStatus === "available")).toBe(true); + // Preparation can reuse its separate projection. Also access the shared + // package itself to prove eviction rechecks use after verification. + await f.store.versionFiles(f.skills[0]!.version); + release(); + expect((await reading)[0]!.content).toBe(changed[0]!.content); + expect((await fs.stat(paths.directory)).isDirectory()).toBe(true); + } finally { clearTimeout(timeout); release(); await Promise.allSettled([reading, warming]); } + }); + + it("bounds cold response lifetimes while cleanup is waiting without blocking warm preparation", async () => { + const fixtures = [await fixture(), await fixture(), await fixture()]; + let bothEntered!: () => void; + const started = new Promise(resolve => { bothEntered = resolve; }); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + let decoded = 0; + const trim = vi.spyOn(packageCache, "trimSkillPackageCache").mockImplementation(async () => { + if (++decoded === 2) bothEntered(); + await blocked; + }); + const reads = fixtures.slice(0, 2).map(f => f.store.versionFiles(f.skills[0]!.version)); + const storageRead = vi.spyOn(storage, "getObject"); + try { + await started; + reads.push(fixtures[2]!.store.versionFiles(fixtures[2]!.skills[0]!.version)); + expect((await fixtures[0]!.resolve()).every(entry => entry.sourceStatus === "available")).toBe(true); + // Let the third metadata/manifest read settle. Its archive and response + // must remain queued while two decoded responses are owned by cleanup. + await new Promise(resolve => setTimeout(resolve, 50)); + expect(decoded).toBe(2); + const third = (await fixtures[2]!.store.descriptor(fixtures[2]!.companyId, fixtures[2]!.id)).storage!; + expect(storageRead.mock.calls.filter(call => call[1] === third.archiveKey)).toHaveLength(0); + release(); + expect((await Promise.all(reads)).every(files => files[0]!.path === "SKILL.md")).toBe(true); + expect(trim).toHaveBeenCalledTimes(3); + } finally { release(); await Promise.allSettled(reads); } + }); + + it.each(["manifest", "files"] as const)("does not load queued legacy bodies for %s reads", async kind => { + const f = await fixture(); + const companyId = await company(); + const rows = await db.insert(snapshots).values(Array.from({ length: 33 }, (_, index) => { + const inventory = files.map((file, i) => i ? file : { ...file, content: `${file.content}${index}` }); + return { companyId, contentHash: skillSnapshotHash(inventory), fileInventory: inventory }; + })).returning({ id: snapshots.id }); + const store = skillPackageStore(db, storage); + if (kind === "files") for (const row of rows) await store.manifest(companyId, row.id); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + let active = 0; + const occupying = Array.from({ length: 2 }, () => f.store.withIsolatedPackage(f.companyId, f.id, async () => { + active++; + await blocked; + })); + await vi.waitFor(() => expect(active).toBe(2)); + let bodies = 0; + const observe = (_connection: number, query: string) => { + if (query.includes('"file_inventory"') && query.includes('"company_skill_repository_snapshots"')) bodies++; + }; + const debug = db.$client.options.debug; + db.$client.options.debug = observe; + const dedicated = dbModule.withDedicatedDbConnection; + let ready = 0; + vi.spyOn(dbModule, "withDedicatedDbConnection").mockImplementation((owner, action) => dedicated(owner, connection => { + connection.$client.options.debug = observe; + if (kind === "files") ready++; + return action(connection); + })); + if (kind === "manifest") { + const parent = skillPackagePaths(companyId, rows[0]!.id).parent; + const lstat = fs.lstat.bind(fs); + vi.spyOn(fs, "lstat").mockImplementation(async (...args) => { + const result = await lstat(...args); + if (args[0] === parent) ready++; + return result; + }); + } + const read = (id: string) => kind === "files" ? store.files(companyId, id, true) : store.manifest(companyId, id); + const queued = rows.slice(0, 32).map(row => read(row.id)); + try { + await vi.waitFor(() => expect(ready).toBe(32)); + // The final reader observes the full queue only after all earlier readers + // have reached admission. None may preload repository bodies while waiting. + await expect(read(rows[32]!.id)).rejects.toThrow("busy"); + expect(bodies).toBe(0); + release(); + expect(await Promise.all(queued)).toHaveLength(32); + } finally { + release(); + await Promise.allSettled([...occupying, ...queued]); + db.$client.options.debug = debug; + } + }, 15_000); + + it("does not deadlock a transaction behind cold reads on a one-connection pool", async () => { + const fixtures = [await fixture(), await fixture(), await fixture()]; + const roots = new Map(fixtures.slice(0, 2).map(f => [skillPackagePaths(f.companyId, f.id).directory, 0])); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + const lstat = fs.lstat.bind(fs); + const stat = vi.spyOn(fs, "lstat").mockImplementation(async (...args) => { + const root = String(args[0]); + if (roots.has(root)) { + roots.set(root, roots.get(root)! + 1); + await blocked; + throw Object.assign(new Error("Cold package"), { code: "ENOENT" }); + } + return lstat(...args); + }); + const reads = fixtures.slice(0, 2).map(f => f.store.versionFiles(f.skills[0]!.version)); + let timeout: ReturnType | undefined; + try { + // Both callers have loaded metadata but have not entered cold fallback. + await vi.waitFor(() => expect([...roots.values()]).toEqual([1, 1])); + await db.transaction(async tx => { + await tx.execute(sql`select 1`); + release(); + // Their second cache misses now queue the cold fallback while this + // transaction owns the only ordinary connection. + await vi.waitFor(() => expect([...roots.values()]).toEqual([2, 2])); + const cancellation = new AbortController(); + timeout = setTimeout(() => cancellation.abort(new Error("Cold read deadlocked the pool")), 1_000); + const third = fixtures[2]!; + expect(await skillPackageStore(tx as unknown as typeof db, storage, db) + .versionFiles(third.skills[0]!.version, true, cancellation.signal)) + .toEqual([{ path: "SKILL.md", kind: "skill", content: files[0]!.content }]); + }); + } finally { + clearTimeout(timeout); + release(); + stat.mockRestore(); + await Promise.allSettled(reads); + } + expect((await Promise.all(reads)).every(result => result[0]!.content === files[0]!.content)).toBe(true); + }); + + it("bounds concurrent cached inventories while leaving warm turn preparation responsive", async () => { + const f = await fixture(); + await f.resolve(); + const target = path.join(skillPackagePaths(f.companyId, f.id).directory, files[0]!.path); + const open = fs.open.bind(fs); + let entered = 0; + let bothEntered!: () => void; + const started = new Promise(resolve => { bothEntered = resolve; }); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + vi.spyOn(fs, "open").mockImplementation(async (...args) => { + if (args[0] === target) { + if (++entered === 2) bothEntered(); + await blocked; + } + return open(...args); + }); + const version = { ...f.skills[0]!.version, packageFileMap: null }; + const storageRead = vi.spyOn(storage, "getObject"); + const reads = [f.store.versionFiles(version), f.store.versionFiles(version)]; + let preview: ReturnType | undefined; + try { + await started; + reads.push(f.store.versionFiles(version)); + preview = f.store.readVersionFile(version, files[0]!.path); + reads.push(...Array.from({ length: 30 }, () => f.store.versionFiles(version))); + // These queries share the one-connection pool, so the third inventory has + // loaded its descriptor by the time preparation finishes. Its byte read + // must still be queued behind the two active readers. + expect((await f.resolve()).every(entry => entry.sourceStatus === "available")).toBe(true); + expect(entered).toBe(2); + await expect(f.store.versionFiles(version)).rejects.toThrow("busy"); + expect(storageRead).not.toHaveBeenCalled(); + release(); + expect(await Promise.all(reads)).toEqual(reads.map(() => files)); + expect(await preview).toMatchObject(files[0]!); + } finally { release(); await Promise.allSettled([...reads, preview]); } + }); + + it("reads cached versions while both storage operations are stalled", async () => { + const f = await fixture(); + await f.resolve(); + const get = storage.getObject.bind(storage); + let entered = 0; + let bothEntered!: () => void; + const started = new Promise(resolve => { bothEntered = resolve; }); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + vi.spyOn(storage, "getObject").mockImplementation(async (...args) => { + if (++entered === 2) bothEntered(); + await blocked; + return get(...args); + }); + const pending = [f.store.withIsolatedPackage(f.companyId, f.id, async () => null), + f.store.withIsolatedPackage(f.companyId, f.id, async () => null)]; + let timeout: ReturnType | undefined; + try { + await started; + const reading = db.transaction(tx => skillPackageStore(tx as unknown as typeof db, storage) + .versionFiles({ ...f.skills[0]!.version, packageFileMap: null })); + expect(await Promise.race([reading, new Promise((_resolve, reject) => { + timeout = setTimeout(() => reject(new Error("Cached read waited for storage")), 1_000); + })])).toEqual(files); + } finally { clearTimeout(timeout); release(); await Promise.allSettled(pending); } + }); + + it("retries expired upload claims with fresh keys and only deletes the abandoned attempt", async () => { + const companyId = await company(); + const store = skillPackageStore(db, storage); + const put = vi.spyOn(storage, "putFile").mockRejectedValueOnce(new Error("connection lost")); + await expect(store.prepare(companyId, files)).rejects.toThrow("connection lost"); + put.mockRestore(); + const [pending] = await db.select().from(snapshots).where(eq(snapshots.companyId, companyId)); + expect(pending!.storage).toBeNull(); + await expect(store.prepare(companyId, files)).rejects.toThrow("being saved"); + await db.update(snapshots).set({ upload: { ...pending!.upload!, expiresAt: new Date(0).toISOString() } }).where(eq(snapshots.id, pending!.id)); + const id = await store.prepare(companyId, files); + expect(id).toBe(pending!.id); + const receipt = (await store.descriptor(companyId, id)).storage!; + expect(receipt.archiveKey).not.toBe(pending!.upload!.archiveKey); + const [deletion] = await db.select().from(skillPackageDeletions).where(eq(skillPackageDeletions.companyId, companyId)); + expect(deletion!.archiveKey).toBe(pending!.upload!.archiveKey); + await db.update(skillPackageDeletions).set({ deleteAfter: new Date(0) }).where(eq(skillPackageDeletions.id, deletion!.id)); + expect((await skillPackageMaintenance(db, storage).deleteObjects({ companyId, apply: true })).removed).toBe(1); + expect(await store.verifyDurable(companyId, id)).toBe(true); + // Company cascades retain a receipt after removing the snapshot's DB row. + await db.delete(companies).where(eq(companies.id, companyId)); + expect(await db.select().from(skillPackageDeletions).where(eq(skillPackageDeletions.companyId, companyId))) + .toEqual([expect.objectContaining({ archiveKey: receipt.archiveKey })]); + }); + + it("cancels queued and active uploads without publishing or leaking I/O capacity", async () => { + const ids = await Promise.all([company(), company(), company()]); + const active = new AbortController(); + const queued = new AbortController(); + const put = vi.spyOn(storage, "putFile").mockImplementation(async input => { + expect(input.signal).toBeDefined(); + await new Promise((_resolve, reject) => { + input.signal!.addEventListener("abort", () => reject(input.signal!.reason), { once: true }); + }); + throw new Error("Unreachable cancelled upload"); + }); + const store = skillPackageStore(db, storage); + const running = ids.slice(0, 2).map(id => store.prepare(id!, files, false, active.signal).catch(error => error)); + try { + await vi.waitFor(() => expect(put).toHaveBeenCalledTimes(2)); + const waiting = store.prepare(ids[2]!, files, false, queued.signal).catch(error => error); + await vi.waitFor(async () => expect(await db.select({ id: snapshots.id }).from(snapshots) + .where(eq(snapshots.companyId, ids[2]!))).toHaveLength(1)); + queued.abort(new Error("queued cancellation")); + expect((await waiting).message).toBe("queued cancellation"); + expect(put).toHaveBeenCalledTimes(2); + } finally { + active.abort(new Error("active cancellation")); + await Promise.all(running); + put.mockRestore(); + } + for (const id of ids) { + const [row] = await db.select().from(snapshots).where(eq(snapshots.companyId, id!)); + expect(row!.storage).toBeNull(); + expect(row!.upload).not.toBeNull(); + } + expect(await store.prepare(await company(), files)).toEqual(expect.any(String)); + }); + + it("migrates both legacy payload locations in bounded resumable pages without changing version IDs", async () => { + const companyId = await company(); + const [snapshot] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const [skill] = await db.insert(companySkills).values({ companyId, key: "legacy", slug: "legacy", name: "Legacy", markdown: files[0]!.content }).returning(); + const original = { path: "SKILL.md", kind: "skill" as const, content: files[0]!.content }; + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 7, + repositorySnapshotId: snapshot!.id, repositorySkillPath: files[0]!.path, fileInventory: [original] }).returning(); + const maintenance = skillPackageMaintenance(db, storage); + expect((await maintenance.migrateSnapshots({ companyId })).migrated).toBe(0); + const fail = vi.spyOn(storage, "getObject").mockRejectedValueOnce(new Error("verification interrupted")); + await expect(maintenance.migrateSnapshots({ companyId, apply: true })).rejects.toThrow("verification interrupted"); + fail.mockRestore(); + const [pending] = await db.select().from(snapshots).where(eq(snapshots.id, snapshot!.id)); + expect(pending!.fileInventory).toEqual(files); + await db.update(snapshots).set({ upload: { ...pending!.upload!, expiresAt: new Date(0).toISOString() } }).where(eq(snapshots.id, snapshot!.id)); + expect((await maintenance.migrateSnapshots({ companyId, limit: 1, apply: true })).migrated).toBe(1); + const [soaked] = await db.select().from(snapshots).where(eq(snapshots.id, snapshot!.id)); + expect(soaked!.storage).not.toBeNull(); + expect(soaked!.fileInventory).toEqual(files); + expect((await maintenance.migrateSnapshots({ companyId, limit: 1, apply: true, clearLegacy: true })).migrated).toBe(1); + expect((await maintenance.migrateSnapshots({ companyId, apply: true })).inspected).toBe(0); + expect((await maintenance.migrateVersions({ companyId, apply: true, clearLegacy: true })).migrated).toBe(1); + const [saved] = await db.select().from(companySkillVersions).where(and(eq(companySkillVersions.id, version!.id), eq(companySkillVersions.companyId, companyId))); + expect(saved).toMatchObject({ id: version!.id, revisionNumber: 7, fileInventory: [], repositorySnapshotId: snapshot!.id, packageSnapshotId: snapshot!.id }); + expect(await skillPackageStore(db, storage).versionFiles(saved!)).toEqual([original]); + }); + + it.each([false, true])("restores across legacy payload clearing with a cached manifest: %s", async cachedManifest => { + const companyId = await company(); + const [snapshot] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const store = skillPackageStore(db, storage); + await withSkillPackageReadScope(db, async () => { + expect((await store.descriptor(companyId, snapshot!.id)).storage).toBeNull(); + if (cachedManifest) await store.manifest(companyId, snapshot!.id); + await store.prepare(companyId, files, true); + await db.update(snapshots).set({ fileInventory: [] }).where(eq(snapshots.id, snapshot!.id)); + expect(skillSnapshotHash(await store.files(companyId, snapshot!.id))).toBe(skillSnapshotHash(files)); + expect(await store.verifyDurable(companyId, snapshot!.id)).toBe(true); + }); + }); + + it("bounds and validates archives, binaries, paths, executable modes, and corruption", async () => { + const root = await fs.mkdtemp(path.join(home, "codec-")); + const archive = path.join(root, "files.tar.gz"); + const manifest = skillPackageManifest(files); + await writeSkillPackageArchive(archive, files); + const output = path.join(root, "out"); + await fs.mkdir(output); + await extractSkillPackageArchive(archive, output, manifest); + expect(await verifySkillPackageDirectory(output, manifest.files)).toBe(true); + expect((await fs.stat(path.join(output, "shared/run.sh"))).mode & 0o111).toBe(0o111); + expect(() => validateSkillPackageManifest({ ...manifest, files: [{ ...manifest.files[0], path: "../escape" }] })).toThrow(); + expect(() => validateSkillPackageManifest({ ...manifest, files: [manifest.files[0], manifest.files[0]] })).toThrow(); + expect(() => validateSkillPackageManifest({ ...manifest, totalBytes: 1 })).toThrow(); + const boundary = { ...manifest, totalBytes: 100 * 1024 * 1024, + files: [{ ...manifest.files[0]!, size: 100 * 1024 * 1024 }] }; + expect(validateSkillPackageManifest(boundary).totalBytes).toBe(boundary.totalBytes); + expect(() => validateSkillPackageManifest({ ...boundary, totalBytes: boundary.totalBytes + 1, + files: [{ ...boundary.files[0]!, size: boundary.totalBytes + 1 }] })).toThrow("byte bound"); + const bad = path.join(root, "bad"); + await fs.mkdir(bad); + await fs.truncate(archive, 20); + await expect(extractSkillPackageArchive(archive, bad, manifest)).rejects.toThrow(); + }); + + it("stores and converts a saved directory package above 64 MiB without losing its history", async () => { + const companyId = await company(); + const content = "x".repeat(1024 * 1024); + const large = [{ path: "SKILL.md", kind: "skill" as const, content: "---\nname: large\n---\nlarge skill" }, + ...Array.from({ length: 65 }, (_, index) => ({ path: `asset-${index}`, kind: "asset" as const, content }))]; + const store = skillPackageStore(db, storage); + const id = await store.prepare(companyId, large); + const [skill] = await db.insert(companySkills).values({ companyId, key: "large", slug: "large", name: "Large", markdown: large[0]!.content }).returning(); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, + revisionNumber: 1, fileInventory: large }).returning(); + await skillPackageMaintenance(db, storage).migrateVersions({ id: version!.id, apply: true, clearLegacy: true }); + const [converted] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version!.id)); + expect(converted!.fileInventory).toEqual([]); + expect(converted!.migrationAt).toBeNull(); + expect(converted!.packageSnapshotId).toBe(id); + expect(skillSnapshotHash(await store.versionFiles(converted!))).toBe(skillSnapshotHash(large)); + }, 90_000); +}); diff --git a/server/src/__tests__/skill-repository-package.test.ts b/server/src/__tests__/skill-repository-package.test.ts index 8c1a602dcf5..91eef93caba 100644 --- a/server/src/__tests__/skill-repository-package.test.ts +++ b/server/src/__tests__/skill-repository-package.test.ts @@ -4,6 +4,7 @@ import os from 'node:os'; import path from 'node:path'; import { materializePaperclipSkillCopy } from '@paperclipai/adapter-utils/server-utils'; import { repositorySkillProjection } from '../services/skill-repository-package.js'; +import { skillPackageManifest } from '../services/skill-package-files.js'; import { snapshotFile, skillFileBytes } from '../services/skill-snapshot.js'; const md = '---\nname: main\ndescription: Work\ndisable-model-invocation: true\n---\nRead [runtime](../runtime/SKILL.md) and ../../agents/worker.md.\n'; @@ -37,6 +38,15 @@ describe('repository skill delivery', () => { expect(await fs.readFile(path.join(sandbox, 'agents/openai.yaml'), 'utf8')).toContain('allow_implicit_invocation: false'); } finally { await fs.rm(root, { recursive: true, force: true }); } }); + it('accepts scanner-limit paths after adding the portable repository prefix', () => { + const deep = [...Array(63).fill('d'), 'SKILL.md'].join('/'); + const long = `${'d'.repeat(4096 - '/SKILL.md'.length)}/SKILL.md`; + for (const name of [deep, long]) { + const original = [{ path: name, kind: 'skill' as const, content: md }]; + expect(() => skillPackageManifest(repositorySkillProjection(original, name))).not.toThrow(); + expect(() => skillPackageManifest([{ ...original[0]!, path: `extra/${name}` }])).toThrow(); + } + }); it('rejects escaped and absent entrypoints', () => { expect(() => repositorySkillProjection([], '../SKILL.md')).toThrow(/Invalid/); expect(() => repositorySkillProjection([], 'missing/SKILL.md')).toThrow(/missing/); diff --git a/server/src/__tests__/skill-sources.test.ts b/server/src/__tests__/skill-sources.test.ts index 7a8ca0d6043..7c72d49f95a 100644 --- a/server/src/__tests__/skill-sources.test.ts +++ b/server/src/__tests__/skill-sources.test.ts @@ -1,3 +1,6 @@ +import * as recovery from '../services/skill-package-recovery.js'; +import { skillPackageStore, skillPackagePaths } from '../services/skill-package-store.js'; +import { getStorageService } from '../storage/index.js'; import { resolveExternalTestDatabaseUrl } from './helpers/external-test-database.js'; import { applyPendingMigrations } from '@paperclipai/db'; import { randomUUID } from 'node:crypto'; @@ -31,7 +34,9 @@ describe.skipIf(!support.supported)('skill source persistence', () => { let files: Record = initial; let commit = sha; const context: SkillSourceContext = { actor: { type: 'user', userId: 'board' }, read: () => githubFixture(files, { 'deep/one/scripts/run.sh': '100755', 'shared/check.sh': '100755' }, commit), authorize: async () => {}, audit: async () => {} }; + const previousPackageStorage = process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; beforeAll(async () => { + process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "true"; if (externalUrl) await applyPendingMigrations(externalUrl); else testDb = await startEmbeddedPostgresTestDatabase('paperclip-skill-sources-'); db = createDb(externalUrl ?? testDb.connectionString); @@ -39,10 +44,24 @@ describe.skipIf(!support.supported)('skill source persistence', () => { await db.insert(companies).values({ id: companyId, name: 'Skills', issuePrefix: 'SKL' }); }, 30000); afterAll(async () => { + if (previousPackageStorage === undefined) delete process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; + else process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = previousPackageStorage; if (originalHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = originalHome; await db?.$client.end({ timeout: 1 }); await testDb?.cleanup(); if (home) await fs.rm(home, { recursive: true, force: true }); }); + it('tracks repositories with no selected valid skills without saving their contents', async () => { + for (const repository of [{ 'README.md': 'No skills here' }, { 'one/SKILL.md': md('one'), 'shared.txt': 'Unused package content' }]) { + const id = randomUUID(); + await db.insert(companies).values({ id, name: 'Discovery only', issuePrefix: `E${id.slice(0, 6)}` }); + const result = await skillSourceService(db).create(id, { repositoryUrl: 'https://github.com/acme/skills', trackingRef: 'main', + packageMode: 'repository', commitSha: sha, selectedPaths: [] }, { ...context, read: () => githubFixture(repository) }); + expect(result.imported).toEqual([]); + expect(result.source.lastScanCommit).toBe(sha); + expect(await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, id))).toEqual([]); + } + }); + it('publishes two assigned skills while a concurrent version save holds the second skill', async () => { const companyId = randomUUID(); await db.insert(companies).values({ id: companyId, name: 'Concurrent skill publication', issuePrefix: 'CSP' }); @@ -296,6 +315,34 @@ describe.skipIf(!support.supported)('skill source persistence', () => { expect((await skillSourceService(db).list(companyId)).some(source => source.trackingRef === 'rollback')).toBe(false); expect((await db.select().from(companySkillVersions).where(eq(companySkillVersions.companyId, companyId))).every(version => before.some(skill => skill.id === version.companySkillId))).toBe(true); }); + it.each(['skills', 'repository'] as const)('binds %s recovery to the committed retry after an import rollback', async packageMode => { + const isolated = randomUUID(); + await db.insert(companies).values({ id: isolated, name: 'Retry recovery', issuePrefix: `R${isolated.slice(0, 6)}` }); + const original = { 'one/SKILL.md': md('recover retry'), 'one/helper.md': 'Original helper' }; + const reader = githubFixture(original, {}, sha); + const input = { repositoryUrl: 'https://github.com/acme/skills', trackingRef: 'main', packageMode, commitSha: sha, selectedPaths: ['one/SKILL.md'] }; + const service = skillSourceService(db); + await expect(service.create(isolated, input, { ...context, read: () => reader, + audit: async () => { throw new Error('failed import transaction'); } })).rejects.toThrow('failed import transaction'); + const [abandoned] = await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, isolated)); + const result = await service.create(isolated, input, { ...context, read: () => reader }); + const [saved] = await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.id, abandoned!.id)); + expect(saved!.restore!.sourceId).toBe(result.source.id); + expect(saved!.restore!.sourceId).not.toBe(abandoned!.restore!.sourceId); + await getStorageService().deleteObject(isolated, saved!.storage!.archiveKey); + await fs.rm(skillPackagePaths(isolated, saved!.id).directory, { recursive: true, force: true }); + const authorize = vi.spyOn(recovery, 'skillPackageRecoveryReader').mockImplementation(async (database, company, recipe) => { + const [source] = await database.select().from(companySkillSources).where(and(eq(companySkillSources.companyId, company), eq(companySkillSources.id, recipe.sourceId))); + expect(source!.id).toBe(result.source.id); + return reader; + }); + try { + const restored = await skillPackageStore(db).files(isolated, saved!.id); + expect(restored.find(file => file.path.endsWith('SKILL.md'))!.content).toBe(original['one/SKILL.md']); + expect(authorize).toHaveBeenCalledOnce(); + } finally { authorize.mockRestore(); } + }); + it('retains excluded folders, reviews additions, restores removed paths and preserves pinned versions and copies', async () => { const service = skillSourceService(db), skills = companySkillService(db); files = { 'keep/SKILL.md': md('keep'), 'keep/assets/data.bin': Buffer.from([0,255]), 'keep/run.sh': '#!/bin/sh\necho original', 'declined/one/SKILL.md': md('one') }; commit = sha; @@ -566,7 +613,7 @@ describe.skipIf(!support.supported)('skill source persistence', () => { const v1 = (await skills.getVersion(companyId, main.id, main.currentVersionId!))!; const helper = installed.imported.find(skill => skill.name === 'helper')!; expect((await skills.getVersion(companyId, helper.id, helper.currentVersionId!))?.repositorySnapshotId).toBe(v1.repositorySnapshotId); - expect(await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, companyId))).toHaveLength(1); + expect(await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.id, v1.repositorySnapshotId!))).toEqual([expect.objectContaining({ fileInventory: [], storage: expect.objectContaining({ format: "tar-gzip-v1" }) })]); await db.update(companySkills).set({ trustLevel: 'markdown_only' }).where(eq(companySkills.id, main.id)); expect((await sources.refresh(companyId, installed.source.id, context)).unchanged).toBe(2); expect((await skills.getById(companyId, main.id))?.trustLevel).toBe('scripts_executables'); diff --git a/server/src/__tests__/storage-local-provider.test.ts b/server/src/__tests__/storage-local-provider.test.ts index 8ff225fcf46..e35500896fa 100644 --- a/server/src/__tests__/storage-local-provider.test.ts +++ b/server/src/__tests__/storage-local-provider.test.ts @@ -1,4 +1,4 @@ -import { afterEach, describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; import { Readable } from "node:stream"; import { createHash } from "node:crypto"; import os from "node:os"; @@ -118,4 +118,17 @@ describe("local disk storage provider", () => { await service.deleteObject("company-1", stored.objectKey); await expect(service.getObject("company-1", stored.objectKey)).rejects.toMatchObject({ status: 404 }); }); + + it.each(["headObject", "deleteObject"] as const)("cancels %s even if a provider stops responding", async method => { + const provider = createLocalDiskStorageProvider(os.tmpdir()); + const request = vi.spyOn(provider, method).mockImplementation(async () => new Promise(() => {})); + const service = createStorageService(provider); + const controller = new AbortController(); + const pending = service[method]("company-1", "company-1/expired", { signal: controller.signal }); + expect(request).toHaveBeenCalledWith({ objectKey: "company-1/expired", signal: controller.signal }); + controller.abort(new Error("maintenance stopped")); + await expect(pending).rejects.toThrow("maintenance stopped"); + await expect(service[method]("company-1", "company-1/expired", { signal: controller.signal })).rejects.toThrow("maintenance stopped"); + expect(request).toHaveBeenCalledOnce(); + }); }); diff --git a/server/src/__tests__/storage-s3-multipart.test.ts b/server/src/__tests__/storage-s3-multipart.test.ts index b3bbd6ae4b7..343bb0ddcd1 100644 --- a/server/src/__tests__/storage-s3-multipart.test.ts +++ b/server/src/__tests__/storage-s3-multipart.test.ts @@ -1,7 +1,10 @@ -import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, S3Client } from "@aws-sdk/client-s3"; +import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMultipartUploadCommand, UploadPartCommand, DeleteObjectCommand, PutObjectCommand, S3Client } from "@aws-sdk/client-s3"; import { Readable } from "node:stream"; import { describe, expect, it, vi } from "vitest"; import { putS3Multipart } from "../storage/s3-multipart.js"; +import { createS3StorageProvider } from "../storage/s3-provider.js"; +import { createStorageService } from "../storage/service.js"; +import { STORAGE_FAILURE_CLEANUP_MS } from "../storage/failure-cleanup.js"; describe("streamed S3 multipart uploads", () => { it("streams exact ordered parts without buffering the full response", async () => { @@ -45,4 +48,35 @@ describe("streamed S3 multipart uploads", () => { expect(body.destroyed).toBe(true); } finally { send.mockRestore(); client.destroy(); } }); + it("bounds stalled multipart abort and object cleanup after cancellation", async () => { + vi.useFakeTimers(); + const cancellation = new AbortController(); + const cleanupSignals: AbortSignal[] = []; + const send = vi.spyOn(S3Client.prototype, "send").mockImplementation(async (command: any, options: any) => { + if (command instanceof CreateMultipartUploadCommand) return { UploadId: "upload" }; + if (command instanceof UploadPartCommand) return new Promise((_resolve, reject) => { + options.abortSignal.addEventListener("abort", () => reject(cancellation.signal.reason), { once: true }); + }); + if (command instanceof AbortMultipartUploadCommand || command instanceof DeleteObjectCommand) { + cleanupSignals.push(options.abortSignal); + return new Promise(() => {}); + } + expect(command).toBeInstanceOf(PutObjectCommand); + return {}; + }); + const storage = createStorageService(createS3StorageProvider({ bucket: "test", region: "us-east-1" })); + const body = Readable.from([Buffer.alloc(17 * 1024 * 1024)]); + try { + const saving = storage.putFile({ companyId: "company", namespace: "skill-packages", objectKey: "company/package/archive", + originalFilename: "archive", contentType: "application/gzip", body, byteSize: 17 * 1024 * 1024, sha256: "a".repeat(64), signal: cancellation.signal }).catch(error => error); + await vi.advanceTimersByTimeAsync(0); + cancellation.abort(new Error("cancelled upload")); + await vi.advanceTimersByTimeAsync(STORAGE_FAILURE_CLEANUP_MS * 2 + 1); + expect((await saving).message).toBe("cancelled upload"); + expect(cleanupSignals).toHaveLength(2); + expect(cleanupSignals.every(signal => signal.aborted)).toBe(true); + expect(body.destroyed).toBe(true); + await expect(storage.putFile({ companyId: "company", namespace: "skill-packages", originalFilename: "next", contentType: "text/plain", body: Buffer.from("next") })).resolves.toMatchObject({ byteSize: 4 }); + } finally { send.mockRestore(); vi.useRealTimers(); } + }); }); diff --git a/server/src/index.ts b/server/src/index.ts index 1c51bfb6a8b..db8b56da233 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -23,6 +23,8 @@ import { registerEnvironmentCleanupWork } from "./services/environment-cleanup-w import { loginCleanupSchedule } from "./services/login-cleanup-state.js"; import { registerLoginCleanupWork } from "./services/login-cleanup-work.js"; import { timeStartupRecovery } from "./startup-recovery-timing.js"; +import { runtimeAssetRetention } from "./services/native-runtime/runtime-asset-retention.js"; +import { skillPackageWork } from "./services/skill-package-work.js"; import { skillSourceSyncService } from "./services/skill-source-sync.js"; import { skillVersionRetentionService, @@ -1378,6 +1380,9 @@ async function startServerWithDatabaseTeardown( const accountingWorker = registerCostAccountingWork(db, deliveryWork, accountingHeartbeat.reconcileCostAccounting); server.on("close", accountingWorker.stop); skillSourceSync.start(deliveryWork); + skillPackageWork(db).start(deliveryWork); + const runtimeAssetCleanup = runtimeAssetRetention(db).start(deliveryWork); + server.on("close", () => { void runtimeAssetCleanup.stop(); }); const caseHandoffDeliveries = caseHandoffDeliveryService(db as any, { heartbeat: environmentLeaseCleanupHeartbeat }); deliveryWork.register(DELIVERY_QUEUES.caseHandoffs, { retryMs: 5_000, diff --git a/server/src/routes/company-skills.ts b/server/src/routes/company-skills.ts index 93a396fcb9c..3926793700a 100644 --- a/server/src/routes/company-skills.ts +++ b/server/src/routes/company-skills.ts @@ -1221,7 +1221,7 @@ export function companySkillRoutes(db: Db) { eq(activityLog.companyId, companyId), eq(activityLog.action, "company.skill_created"), sql`${activityLog.details}->>'idempotencyKey' = ${receiptKey}`, )); - const service = companySkillService(tx as unknown as Db); + const service = companySkillService(tx as unknown as Db, { packageDb: db }); if (prior) { if (prior.details?.fingerprint !== fingerprint) throw conflict("Skill idempotency key was used with different inputs"); const skill = await service.getById(companyId, prior.entityId); @@ -1298,7 +1298,7 @@ export function companySkillRoutes(db: Db) { } let publication: ActivityPublication | null = null; let receipt: { skillId: string; path: string; versionId: string | null; studioPath: string } | null = null; - await companySkillService(tx as unknown as Db).updateFile(companyId, skillId, input.path, input.content, skillActor(req), { + await companySkillService(tx as unknown as Db, { packageDb: db }).updateFile(companyId, skillId, input.path, input.content, skillActor(req), { encoding: input.encoding, executable: input.executable, expectedVersionId, onRollback: (restore, versionId) => { rollbackState.current = { restore, versionId }; }, afterUpdate: async (versionId) => { diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index 7a9b0e55052..7ab88aadfa9 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -74,6 +74,7 @@ export async function deleteCompany(db: Db, id: string) { await notifyChatCompanyConfiguration(tx, id); await chatCompanyDeletion.deleteCompanyData(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); await notifyDeliveryWork(tx, DELIVERY_QUEUES.routineSchedules); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyDeliveryWork(tx, DELIVERY_QUEUES.secretProposalExpiry, id); diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index b43223f69c6..7ba950039e9 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -1,9 +1,14 @@ import { notifySkillVersionRetention } from "./skill-version-retention-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; +import { verifyFileManifest } from "./file-manifest-cache.js"; +import { skillPackageStorageEnabled, skillPackageStore, skillVersionFileMap, skillVersionMetadataColumns, withSkillPackageReadScope, type StoredSkillVersion } from "./skill-package-store.js"; +import { skillPackageRuntime } from "./skill-package-runtime.js"; +import type { SkillPackageFileMapping } from "@paperclipai/db"; import { ensureSkillCacheDirectory, touchSkillVersionCache, withSkillVersionCacheLock } from "./skill-version-cache.js"; import { measureRunPhase } from "./execution-preparation-timing.js"; import { verifyExistingBundledSkillRelease } from "./bundled-skill-release-validation.js"; import { retainProjectSkillAccess, skillAccessService } from "./skill-access.js"; +import { assertLocalImportOutsideSkillPackageCache } from "./skill-local-import-boundary.js"; import type { AuthorizationActor } from "./authorization.js"; import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js"; import { logger } from "../middleware/logger.js"; @@ -26,8 +31,9 @@ import { companySkillTestInputs, companySkillTestRunTemplates, companySkillTestRuns, - companySkillVersions, companySkillRepositorySnapshots, + companySkillVersions, withDedicatedDbConnection, companySkills, + companySkillRepositorySnapshots, costEvents, documents, issueAttachments, @@ -102,6 +108,7 @@ import type { CompanySkillUpdateHoldReason, CompanySkillUsageAgent, CompanySkillVersion, + CompanySkillVersionSummary, CompanySkillVersionCreateRequest, CompanySkillVersionFileInventoryEntry, IssueAttachment, @@ -408,6 +415,8 @@ type BundledSkillReleaseManifestEntry = { type CreateVersionOptions = { repositorySnapshotId?: string; repositorySkillPath?: string; + packageSnapshotId?: string; + packageFileMap?: SkillPackageFileMapping[] | null; fileInventory?: CompanySkillVersionFileInventoryEntry[]; release?: { id: string; name: string; releasedAt: Date }; updateCurrentVersion?: boolean; @@ -1931,8 +1940,9 @@ function serializeVersionFileInventory( } function toCompanySkillVersion(row: CompanySkillVersionRow): CompanySkillVersion { + const { packageSnapshotId: _package, packageFileMap: _map, migrationAt: _migration, ...publicRow } = row; return { - ...row, + ...publicRow, label: row.label ?? null, releaseId: row.releaseId ?? null, releaseName: row.releaseName ?? null, @@ -2796,7 +2806,7 @@ function enrichSkill( skill: CompanySkill, attachedAgentCount: number, usedByAgents: CompanySkillUsageAgent[] = [], - currentVersion: CompanySkillVersion | null = null, + currentVersion: CompanySkillVersionSummary | null = null, starredByCurrentActor = false, existingForks: CompanySkillForkSummary[] = [], ) { @@ -2966,7 +2976,12 @@ async function listLastEditorsBySkillId( return editors; } -export function companySkillService(db: Db, options: { actor?: AuthorizationActor } = {}) { +export function companySkillService(db: Db, options: { actor?: AuthorizationActor; packageDb?: Db } = {}) { + const packageDb = options.packageDb ?? db; + const packages = skillPackageStore(packageDb); + const packageRuntime = skillPackageRuntime(packageDb); + // Admin reads keep SQL on the caller's transaction, which may own the pool's only slot. + const packageReader = db === packageDb ? packages : skillPackageStore(db, undefined, packageDb); const visibility = (database: DbOrTransaction = db) => skillAccessService(database as Db, options.actor); const folderSvc = folderService(db); const agents = agentService(db); @@ -2979,6 +2994,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo code: "skill_source_validation_failed", }); } + await assertLocalImportOutsideSkillPackageCache(sourceRealPath, resolveManagedSkillsRoot(companyId)); const projectRows = await projects.list(companyId); const configuredRoots = [ resolveManagedSkillsRoot(companyId), @@ -3231,6 +3247,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo await removeRuntimeSkillCache(resolveManagedSkillsRoot(companyId), skill.id, async () => { await fs.rm(resolveRuntimeSkillMaterializedPath(companyId, skill), { recursive: true, force: true }); await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); await notifySkillVersionRetention(tx, companyId); await tx.delete(companySkills).where(eq(companySkills.id, skill.id)); }); @@ -3387,12 +3404,12 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return readFullSkillRows(companyId); } - async function readFullSkillRows(companyId: string): Promise { - if (!visibility().canReadCompany(companyId)) return []; + async function readFullSkillRows(companyId: string, keys?: Set, unrestrictedOnly = false): Promise { + if (!visibility().canReadCompany(companyId) || keys?.size === 0) return []; const rows = await db .select(selectCompanySkillColumns()) .from(companySkills) - .where(and(eq(companySkills.companyId, companyId), await visibility().condition(companyId))) + .where(and(eq(companySkills.companyId, companyId), keys ? inArray(companySkills.key, [...keys]) : undefined, unrestrictedOnly ? isNull(companySkills.accessSourceId) : await visibility().condition(companyId))) .orderBy(asc(companySkills.name), asc(companySkills.key)); return rows.map((row) => toCompanySkill(row)); } @@ -3467,22 +3484,29 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ?? await getByKey(companyId, ref); } - async function getVersion(companyId: string, skillId: string, versionId: string): Promise { - if (!await getById(companyId, skillId)) return null; - const row = await db - .select() - .from(companySkillVersions) - .where(and( - eq(companySkillVersions.companyId, companyId), - eq(companySkillVersions.companySkillId, skillId), - eq(companySkillVersions.id, versionId), - )) - .then((rows) => rows[0] ?? null); - return row ? toCompanySkillVersion(row) : null; + async function readVersionMetadata(companyId: string, skillId: string, versionId: string): Promise { + if (db === packageDb) return packages.version(companyId, skillId, versionId); + const [row] = await db.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.companySkillId, skillId), eq(companySkillVersions.id, versionId))); + return row ?? null; } - async function getCurrentVersion(skill: CompanySkill): Promise { - return skill.currentVersionId ? getVersion(skill.companyId, skill.id, skill.currentVersionId) : null; + async function getVersion(companyId: string, skillId: string, versionId: string): Promise { + if (!await getById(companyId, skillId)) return null; + const row = await readVersionMetadata(companyId, skillId, versionId); + return row ? toCompanySkillVersion({ ...row, fileInventory: await packageReader.versionFiles(row) }) : null; + } + + async function versionSummary(row: StoredSkillVersion): Promise { + return { ...toCompanySkillVersion(row), fileInventory: await packageReader.versionInventory(row) }; + } + async function getVersionSummary(companyId: string, skillId: string, versionId: string) { + if (!await getById(companyId, skillId)) return null; + const row = await readVersionMetadata(companyId, skillId, versionId); + return row ? versionSummary(row) : null; + } + async function getCurrentVersion(skill: CompanySkill): Promise { + return skill.currentVersionId ? getVersionSummary(skill.companyId, skill.id, skill.currentVersionId) : null; } async function isStarredByActor(companyId: string, skillId: string, actor: SkillActor | null | undefined) { @@ -3548,14 +3572,15 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash && skill.currentVersionId) { const versionId = skill.currentVersionId; return withSkillVersionCacheLock(db, companyId, async () => { - const version = await getVersion(companyId, skill.id, versionId); + const version = await readVersionMetadata(companyId, skill.id, versionId); if (!version) throw unprocessable("Installed skill version is unavailable."); const sourceLocator = await materializeVersionSnapshot(companyId, skill, version); const audit = await auditInstalledSkillBytes({ ...skill, sourceType: "local_path", sourceLocator, - fileInventory: await runtimeVersionInventory(companyId, version), + fileInventory: version.repositorySnapshotId || version.packageSnapshotId + ? await packageRuntime.inventory(version) : await packages.versionFiles(version), }); await persistAuditMetadata(skill, audit); return audit; @@ -3624,7 +3649,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const existingForks = await existingForkSummaries(companyId, skill.id, actor); const currentVersion = await getCurrentVersion(skill); const repositoryInventory = currentVersion?.repositorySnapshotId - ? await runtimeVersionInventory(companyId, currentVersion) : null; + ? await packageRuntime.inventory((await readVersionMetadata(companyId, skill.id, currentVersion.id))!) : null; return enrichSkill( repositoryInventory ? { ...skill, fileInventory: repositoryInventory.map(({ path, kind }) => ({ path, kind })) } : skill, usedByAgents.length, @@ -3672,15 +3697,17 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return out; } - async function listVersions(companyId: string, skillId: string): Promise { + async function listVersions(companyId: string, skillId: string): Promise { const skill = await getById(companyId, skillId); if (!skill) throw notFound("Skill not found"); const rows = await db - .select() + .select(skillVersionMetadataColumns) .from(companySkillVersions) .where(and(eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.companySkillId, skillId))) .orderBy(desc(companySkillVersions.revisionNumber)); - return rows.map((row) => toCompanySkillVersion(row)); + const result: CompanySkillVersionSummary[] = []; + for (const row of rows) result.push(await versionSummary(row)); + return result; } async function createVersion( @@ -3696,8 +3723,25 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const fileInventory = serializeVersionFileInventory( options.fileInventory ?? await collectVersionFileInventory(companyId, skill), ); - const previousVersion = options.fileInventory === undefined ? await getCurrentVersion(skill) : null; + const previousVersion = options.fileInventory === undefined && skill.currentVersionId ? await readVersionMetadata(companyId, skill.id, skill.currentVersionId) : null; + // Shipped release rows retain their existing validation contract. All user + // snapshots use the package store; source imports prepare before their transaction. + let packageFileMap = options.packageFileMap ?? null; + const preparePackage = async (store: ReturnType) => { + const id = await store.prepare(companyId, fileInventory); + packageFileMap = skillVersionFileMap(fileInventory, await store.manifest(companyId, id)); + return id; + }; + const packageSnapshotId = options.packageSnapshotId ?? (options.release || !skillPackageStorageEnabled() ? null : options.database + ? await withDedicatedDbConnection(packageDb, connection => preparePackage(skillPackageStore(connection))) + : await preparePackage(packages)); + if (options.packageSnapshotId) { + const metadata = new Map(fileInventory.map(file => [file.path, file])); + packageFileMap = (packageFileMap ?? fileInventory.map(file => ({ path: file.path, kind: file.kind, sourcePath: file.path }))) + .map(file => ({ ...file, encoding: metadata.get(file.path)?.encoding ?? "utf8" })); + } const persist = async (tx: DbOrTransaction) => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); await notifySkillVersionRetention(tx, companyId); await tx.execute(sql` select ${companySkills.id} @@ -3706,6 +3750,14 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo and ${companySkills.companyId} = ${companyId} for update `); + // File edits already own the skill lock when a dedicated connection saves + // their package. Never hold a package lock while waiting for that skill. + if (packageSnapshotId) { + const [saved] = await tx.select({ storage: companySkillRepositorySnapshots.storage }) + .from(companySkillRepositorySnapshots).where(and(eq(companySkillRepositorySnapshots.companyId, companyId), + eq(companySkillRepositorySnapshots.id, packageSnapshotId))).for("update"); + if (!saved?.storage) throw conflict("Skill package changed during publication; retry the operation."); + } const [{ nextRevision }] = await tx .select({ nextRevision: sql`coalesce(max(${companySkillVersions.revisionNumber}), 0) + 1`, @@ -3719,12 +3771,15 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo companySkillId: skillId, repositorySnapshotId: options.repositorySnapshotId ?? previousVersion?.repositorySnapshotId ?? null, repositorySkillPath: options.repositorySkillPath ?? previousVersion?.repositorySkillPath ?? null, + packageSnapshotId, + migrationAt: packageSnapshotId || options.release ? null : new Date(), + packageFileMap, revisionNumber: Number(nextRevision ?? 1), label: input.label?.trim() || null, releaseId: options.release?.id ?? null, releaseName: options.release?.name ?? null, releasedAt: options.release?.releasedAt ?? null, - fileInventory, + fileInventory: packageSnapshotId ? [] : fileInventory, authorAgentId: actor?.type === "agent" ? actor.agentId ?? null : null, authorUserId: actor?.type === "user" ? actor.userId ?? null : null, }) @@ -3741,7 +3796,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo }; const versionRow = options.database ? await persist(options.database) : await db.transaction(persist); if (!versionRow) throw notFound("Failed to persist skill version"); - return toCompanySkillVersion(versionRow); + return toCompanySkillVersion({ ...versionRow, fileInventory }); } async function refreshStarCount(companyId: string, skillId: string) { @@ -4008,6 +4063,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo async function cleanupFailedFork(companyId: string, sourceSkillId: string, forkSkillId: string, forkDir: string) { await db.transaction(async (tx) => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); await notifySkillVersionRetention(tx, companyId); await tx .update(companySkills) @@ -4414,11 +4470,14 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo async function readLoadedSkillFile(skill: CompanySkill, relativePath: string): Promise { const normalizedPath = normalizePortablePath(relativePath || "SKILL.md"); if (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash && skill.currentVersionId) { - const version = await getCurrentVersion(skill); - // Keep original package-relative reads working, and expose the repository - // files from the same saved snapshot used by runtime delivery. - const file = version?.fileInventory.find(entry => entry.path === normalizedPath) - ?? (version?.repositorySnapshotId ? (await runtimeVersionInventory(skill.companyId, version)).find(entry => entry.path === normalizedPath) : undefined); + const version = await readVersionMetadata(skill.companyId, skill.id, skill.currentVersionId); + let file: CompanySkillVersionFileInventoryEntry | undefined; + if (version) { + file = await packageReader.readVersionFile(version, normalizedPath); + if (!file && version.repositorySnapshotId && version.repositorySkillPath) { + file = (await runtimeVersionInventory(skill.companyId, toCompanySkillVersion(version))).find(file => file.path === normalizedPath); + } + } if (!file) throw notFound("Installed skill snapshot is missing a file."); return { ...file, skillId: skill.id, language: file.encoding === "base64" ? null : inferLanguageFromPath(normalizedPath), markdown: file.encoding !== "base64" && isMarkdownPath(normalizedPath), editable: false }; @@ -4832,63 +4891,117 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo input: CompanySkillFileDeleteRequest, actor: SkillActor | null = null, ): Promise { - return withSkillFileMutation(companyId, skillId, async (skill, tx) => { + let backup: { directory: string; source: string; copy: string } | undefined; + let mutation: { slug: string; sourceLocator: string | null; previousVersionId: string | null; savedVersionId?: string } | undefined; + let needsRestore = false; + async function restore() { + if (!backup || !needsRestore) return; + await fs.cp(backup.copy, backup.source, { recursive: true, preserveTimestamps: true, verbatimSymlinks: true }); + needsRestore = false; + } + try { + const result = await withSkillFileMutation(companyId, skillId, async (skill, tx) => { + const source = deriveSkillSourceInfo(skill); + if (!source.editable || skill.sourceType !== "local_path") { + throw unprocessable(source.editableReason ?? "This skill cannot be edited."); + } - const source = deriveSkillSourceInfo(skill); - if (!source.editable || skill.sourceType !== "local_path") { - throw unprocessable(source.editableReason ?? "This skill cannot be edited."); - } + const normalizedPath = normalizePortablePath(input.path); + if (!normalizedPath) { + throw unprocessable("Skill file path is required."); + } - const normalizedPath = normalizePortablePath(input.path); - if (!normalizedPath) { - throw unprocessable("Skill file path is required."); - } + const deletedPaths = input.target === "folder" + ? skill.fileInventory + .map((entry) => normalizePortablePath(entry.path)) + .filter((entryPath) => entryPath.startsWith(`${normalizedPath}/`)) + : skill.fileInventory + .map((entry) => normalizePortablePath(entry.path)) + .filter((entryPath) => entryPath === normalizedPath); - const deletedPaths = input.target === "folder" - ? skill.fileInventory - .map((entry) => normalizePortablePath(entry.path)) - .filter((entryPath) => entryPath.startsWith(`${normalizedPath}/`)) - : skill.fileInventory - .map((entry) => normalizePortablePath(entry.path)) - .filter((entryPath) => entryPath === normalizedPath); - - if (deletedPaths.length === 0) { - throw notFound(input.target === "folder" ? "Skill folder not found" : "Skill file not found"); - } - if (deletedPaths.includes("SKILL.md")) { - throw unprocessable("SKILL.md cannot be deleted."); - } - - const absolutePath = resolveLocalSkillFilePath(skill, normalizedPath); - if (!absolutePath) throw notFound("Skill file not found"); - - await fs.rm(absolutePath, { - recursive: input.target === "folder", - force: false, - }).catch((error) => { - if ((error as NodeJS.ErrnoException).code === "ENOENT") { + if (deletedPaths.length === 0) { throw notFound(input.target === "folder" ? "Skill folder not found" : "Skill file not found"); } - throw error; - }); + if (deletedPaths.includes("SKILL.md")) { + throw unprocessable("SKILL.md cannot be deleted."); + } - await tx - .update(companySkills) - .set({ updatedAt: new Date() }) - .where(eq(companySkills.id, skill.id)); + const absolutePath = resolveLocalSkillFilePath(skill, normalizedPath); + if (!absolutePath) throw notFound("Skill file not found"); - const inventory = await refreshEditedSkillInventory(skill, tx); - await createVersion(companyId, skillId, { - label: input.target === "folder" ? `Deleted ${normalizedPath}/` : `Deleted ${normalizedPath}`, - }, actor, { database: tx, skipInventoryRefresh: true, skill, fileInventory: inventory }); + // Keep the entire subtree, including unlisted files and empty folders. + // A temporary directory outside the skill also supports imported roots + // on other filesystems without adding the backup to its inventory. + const directory = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-skill-delete-")); + backup = { directory, source: absolutePath, copy: path.join(directory, "source") }; + mutation = { slug: skill.slug, sourceLocator: skill.sourceLocator, previousVersionId: skill.currentVersionId }; + await fs.cp(absolutePath, backup.copy, { recursive: true, preserveTimestamps: true, verbatimSymlinks: true }) + .catch((error) => { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + throw notFound(input.target === "folder" ? "Skill folder not found" : "Skill file not found"); + } + throw error; + }); - return { - skillId: skill.id, - path: normalizedPath, - target: input.target, - deletedPaths, - }; - }); + try { + needsRestore = true; + await fs.rm(absolutePath, { recursive: input.target === "folder", force: false }).catch(error => { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + throw notFound(input.target === "folder" ? "Skill folder not found" : "Skill file not found"); + } + throw error; + }); + await tx + .update(companySkills) + .set({ updatedAt: new Date() }) + .where(eq(companySkills.id, skill.id)); + + const inventory = await refreshEditedSkillInventory(skill, tx); + const version = await createVersion(companyId, skillId, { + label: input.target === "folder" ? `Deleted ${normalizedPath}/` : `Deleted ${normalizedPath}`, + }, actor, { database: tx, skipInventoryRefresh: true, skill, fileInventory: inventory }); + mutation.savedVersionId = version.id; + + return { skillId: skill.id, path: normalizedPath, target: input.target, deletedPaths }; + } catch (error) { + // Restore before releasing the mutation lock so a queued edit cannot + // observe missing files or be overwritten by rollback. + await restore(); + throw error; + } + }); + needsRestore = false; + return result; + } catch (error) { + if (needsRestore && mutation) { + const attempted = mutation; + try { + // A lost commit reply does not prove rollback. Reacquire the same + // locks before checking its outcome or touching the source again. + await db.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`${companyId}:${attempted.slug}`}, 0))`); + const [current] = await tx.select({ slug: companySkills.slug, sourceLocator: companySkills.sourceLocator, + currentVersionId: companySkills.currentVersionId }).from(companySkills) + .where(and(eq(companySkills.companyId, companyId), eq(companySkills.id, skillId))).for("update"); + if (current?.slug !== attempted.slug || current.sourceLocator !== attempted.sourceLocator) return; + if (current.currentVersionId === attempted.savedVersionId) needsRestore = false; + else if (current.currentVersionId === attempted.previousVersionId) await restore(); + }); + } catch (recoveryError) { + logger.warn({ err: recoveryError, companyId, skillId }, "Skill deletion transaction outcome could not be checked"); + } + if (needsRestore) { + logger.warn({ companyId, skillId, backupDirectory: backup?.directory }, "Skill deletion backup retained; outcome is unknown or skill changed"); + } + } + throw error; + } finally { + if (backup && !needsRestore) { + await fs.rm(backup.directory, { recursive: true, force: true }).catch(error => { + logger.warn({ err: error, companyId, skillId }, "Skill file deletion backup cleanup failed"); + }); + } + } } async function installUpdate(companyId: string, skillId: string, options: { force?: boolean } = {}): Promise { @@ -6024,16 +6137,16 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return true; } - async function runtimeVersionInventory(companyId: string, version: CompanySkillVersion) { - if (!version.repositorySnapshotId) return version.fileInventory; - const snapshotId = version.repositorySnapshotId; - const [snapshot] = await measureRunPhase("skills.repository.read", () => db.select().from(companySkillRepositorySnapshots).where(and( - eq(companySkillRepositorySnapshots.companyId, companyId), eq(companySkillRepositorySnapshots.id, snapshotId)))); - if (!snapshot || !version.repositorySkillPath) throw unprocessable("Installed repository snapshot is unavailable."); - return repositorySkillProjection(snapshot.fileInventory, version.repositorySkillPath); + async function runtimeVersionInventory(companyId: string, version: CompanySkillVersionSummary) { + if (!version.repositorySnapshotId) { + const row = (await readVersionMetadata(companyId, version.companySkillId, version.id))!; + return packageReader.versionFiles(row); + } + if (!version.repositorySkillPath) throw unprocessable("Installed repository snapshot is unavailable."); + return repositorySkillProjection(await packageReader.files(companyId, version.repositorySnapshotId, true), version.repositorySkillPath); } - async function materializeVersionSnapshot(companyId: string, skill: CompanySkill, version: CompanySkillVersion): Promise { + async function materializeVersionSnapshot(companyId: string, skill: CompanySkill, version: StoredSkillVersion): Promise { return withSkillVersionCacheLock(db, companyId, async () => { const directory = await materializeVersionSnapshotLocked(companyId, skill, version); await touchSkillVersionCache(companyId, skill.id, version.id); @@ -6041,11 +6154,12 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo }); } - async function materializeVersionSnapshotLocked(companyId: string, skill: CompanySkill, version: CompanySkillVersion): Promise { - version = { ...version, fileInventory: await runtimeVersionInventory(companyId, version) }; + async function materializeVersionSnapshotLocked(companyId: string, skill: CompanySkill, version: StoredSkillVersion): Promise { const runtimeRoot = path.resolve(resolveManagedSkillsRoot(companyId), "__versions__", skill.id); const skillDir = path.join(runtimeRoot, version.id); return runVersionSnapshotFlight(skillDir, async () => { + if (version.repositorySnapshotId || version.packageSnapshotId) return packageRuntime.materialize(version, skillDir); + version = { ...version, fileInventory: await packages.versionFiles(version) }; await ensureSkillCacheDirectory(runtimeRoot, true); if (await measureRunPhase("skills.version.verify", () => materializedVersionSnapshotMatches(skillDir, version), { fileCount: version.fileInventory.length })) return skillDir; const staging = await fs.mkdtemp(path.join(runtimeRoot, `.${version.id}-`)); @@ -6088,7 +6202,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ?? (options.snapshotLatest || (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash) || skill.accessSourceId ? skill.currentVersionId : null); if (selectedVersionId) { const versionPath = path.resolve(resolveManagedSkillsRoot(companyId), "__versions__", skill.id, selectedVersionId); - const version = await measureRunPhase("skills.version.read", () => getVersion(companyId, skill.id, selectedVersionId)); + const version = await measureRunPhase("skills.version.read", () => readVersionMetadata(companyId, skill.id, selectedVersionId)); if (!version) { return { status: "missing", @@ -6192,47 +6306,62 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ): Promise { if (options.onlyKeys?.size === 0 || (!options.onlyKeys && options.selectionConfig && !options.includeLegacy && readPaperclipSkillSyncPreference(options.selectionConfig).desiredSkills.length === 0)) return consume([]); - // Installed selections need current authorization, not a company-wide import on every turn. - const selected = options.onlyKeys ?? (options.selectionConfig && !options.includeLegacy + // Probe only the selected metadata. Public selections avoid scanning + // unrelated source permissions; restricted selections still authorize now. + const requested = options.onlyKeys ?? (options.selectionConfig && !options.includeLegacy ? new Set(readPaperclipSkillSyncPreference(options.selectionConfig).desiredSkills) : null); - const installed = selected ? await db.select({ key:companySkills.key, slug:companySkills.slug, - sourceLocator:companySkills.sourceLocator, - sourceKind:sql`${companySkills.metadata}->>'sourceKind'`, - }).from(companySkills).where(and(eq(companySkills.companyId,companyId), - inArray(companySkills.key,[...selected]),await visibility().condition(companyId))) : []; - if (!selected || installed.length !== selected.size) { + const columns = { key: companySkills.key, slug: companySkills.slug, sourceLocator: companySkills.sourceLocator, + sourceKind: sql`${companySkills.metadata}->>'sourceKind'`, accessSourceId: companySkills.accessSourceId }; + let installed = requested?.size && visibility().canReadCompany(companyId) ? await db.select(columns) + .from(companySkills).where(and(eq(companySkills.companyId, companyId), inArray(companySkills.key, [...requested]))) : []; + if (requested && installed.some(skill => skill.accessSourceId !== null)) { + installed = await db.select(columns).from(companySkills).where(and(eq(companySkills.companyId, companyId), + inArray(companySkills.key, [...requested]), await visibility().condition(companyId))); + } + const preparedSelection = requested && installed.length === requested.size; + if (!preparedSelection) { await measureRunPhase("skills.inventory", () => ensureSkillInventoryCurrent(companyId)); } else { for (const skill of installed.filter(skill => skill.sourceKind === "paperclip_bundled")) { const candidates = skill.key === ONBOARDING_FIRST_TASK_SKILL_KEY ? [fileURLToPath(new URL("../onboarding-assets/first-task/skills/first-task/", import.meta.url))] - : resolveBundledSkillsRoot().map(root => path.join(root,skill.slug)); + : resolveBundledSkillsRoot().map(root => path.join(root, skill.slug)); for (const directory of candidates) { - if (!(await fs.stat(path.join(directory,"SKILL.md")).catch(() => null))?.isFile()) continue; + if (!(await fs.stat(path.join(directory, "SKILL.md")).catch(() => null))?.isFile()) continue; if (normalizeSourceLocatorDirectory(skill.sourceLocator) !== path.resolve(directory)) { - const imported = await readLocalSkillImportFromDirectory(companyId,directory,{metadata:{sourceKind:"paperclip_bundled"}}); + const imported = await readLocalSkillImportFromDirectory(companyId, directory, { metadata: { sourceKind: "paperclip_bundled" } }); if (imported.key !== skill.key) throw new Error("Bundled runtime skill key changed"); - await upsertImportedSkills(companyId,[imported],db,{includeFolderPath:false}); + await upsertImportedSkills(companyId, [imported], db, { includeFolderPath: false }); } break; } } } - return withSkillVersionCacheLock(db, companyId, async () => { - const entries = await listRuntimeSkillEntriesLocked(companyId, options); + return withSkillPackageReadScope(db, () => withSkillVersionCacheLock(db, companyId, async () => { + // Recheck scope on the full row read: a public-to-project change between + // the probe and consumption must fail closed. + const runtimeOptions = preparedSelection ? { ...options, onlyKeys: requested } : options; + const entries = await listRuntimeSkillEntriesLocked(companyId, runtimeOptions, + Boolean(preparedSelection && installed.every(skill => skill.accessSourceId === null))); return measureRunPhase("skills.consume", () => consume(entries)); - }); + })); } async function listRuntimeSkillEntriesLocked( companyId: string, options: RuntimeSkillEntryOptions, + unrestrictedOnly = false, ): Promise { - const skills = await measureRunPhase("skills.rows", () => readFullSkillRows(companyId)); + const references = !options.onlyKeys && options.selectionConfig ? await listReferenceTargets(companyId) : []; const selectedKeys = options.onlyKeys ?? (options.selectionConfig ? new Set( (options.includeLegacy ? resolveLegacyPaperclipDesiredSkillNames : resolvePaperclipDesiredSkillNames)( - options.selectionConfig, skills.map(skill => ({ key: skill.key, runtimeName: buildSkillRuntimeName(skill.key, skill.slug) }))), - ) : null); + options.selectionConfig, references.map(skill => ({ key: skill.key, runtimeName: buildSkillRuntimeName(skill.key, skill.slug) }))), + ) : undefined); + const skills = await measureRunPhase("skills.rows", () => readFullSkillRows(companyId, selectedKeys, unrestrictedOnly)); + await packages.prefetch(companyId, skills.flatMap(skill => { + const id = options.versionSelections?.get(skill.key) ?? skill.currentVersionId; + return id ? [id] : []; + })); const out: PaperclipSkillEntry[] = []; for (let skill of skills) { @@ -6241,7 +6370,12 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const unpinned = !options.versionSelections?.get(skill.key); const installedDirectory = options.snapshotLatest && unpinned && (skill.sourceType === "local_path" || skill.sourceType === "catalog") ? await resolveExistingSkillDirectory(normalizeSkillDirectory(skill)) : null; - if (options.snapshotLatest && unpinned && (!skill.currentVersionId || installedDirectory)) { + const installedVersion = installedDirectory && skill.currentVersionId + ? await readVersionMetadata(companyId, skill.id, skill.currentVersionId) : null; + const installedUnchanged = installedDirectory && installedVersion?.packageSnapshotId && !installedVersion.packageFileMap + ? await verifyFileManifest(installedDirectory, (await packages.manifest(companyId, installedVersion.packageSnapshotId)).files) + : false; + if (!installedUnchanged && options.snapshotLatest && unpinned && (!skill.currentVersionId || installedDirectory)) { // Capture installed bytes only. Older remote imports can have no // version yet; execution must not fetch their missing files upstream. const installed: RuntimeSkillSourceResolution | null = installedDirectory ? { status: "available", source: installedDirectory } @@ -6251,8 +6385,11 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo : skill.fileInventory.length === 1 && skill.fileInventory[0]?.path === "SKILL.md" && skill.markdown ? [{ path: "SKILL.md", kind: "skill" as const, content: skill.markdown }] : null; if (inventory) { - const previous = skill.currentVersionId ? await getVersion(companyId, skill.id, skill.currentVersionId) : null; - if (!previous || skillSnapshotHash(previous.fileInventory) !== skillSnapshotHash(inventory)) { + const previous = installedVersion ?? (skill.currentVersionId ? await readVersionMetadata(companyId, skill.id, skill.currentVersionId) : null); + const previousHash = previous?.packageSnapshotId && !previous.packageFileMap + ? (await packages.descriptor(companyId, previous.packageSnapshotId)).contentHash + : previous ? skillSnapshotHash(await packages.versionFiles(previous)) : null; + if (previousHash !== skillSnapshotHash(inventory)) { const version = await createVersion(companyId, skill.id, { label: "Execution snapshot" }, { type: "system" }, { skipInventoryRefresh: true, skill, fileInventory: inventory }); skill = { ...skill, currentVersionId: version.id }; @@ -6774,7 +6911,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo throw unprocessable("Cannot run a skill test for a skill with zero files."); } - const currentVersion = await getCurrentVersion(skill); + const currentVersion = skill.currentVersionId ? await getVersion(companyId, skill.id, skill.currentVersionId) : null; if (!currentVersion || !versionInventorySnapshotEqual(currentVersion.fileInventory, currentSnapshot)) { return createVersion(companyId, skill.id, { label: "Auto version for test run" }, actor); } @@ -7021,7 +7158,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (!run) return null; const harnessIssueGone = Boolean(row.harnessIssueDeletedAt); const [version, issue, documentRows, interactionRows, attachmentRows, workProductRows] = await Promise.all([ - getVersion(companyId, skillId, row.skillVersionId), + getVersionSummary(companyId, skillId, row.skillVersionId), harnessIssueGone ? Promise.resolve(null) : db @@ -7399,6 +7536,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } try { return await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); if (initial.length) await notifySkillVersionRetention(tx, companyId); await beforeDelete?.(tx); const deleted: CompanySkill[] = []; diff --git a/server/src/services/customer-success-inspection.ts b/server/src/services/customer-success-inspection.ts index 2f9a031e82a..c4a5504ed97 100644 --- a/server/src/services/customer-success-inspection.ts +++ b/server/src/services/customer-success-inspection.ts @@ -19,6 +19,7 @@ import { } from "./workspace-file-resources.js"; import { deriveBundleState } from "./agent-instructions.js"; import { readInstructionBytes } from "./agent-instruction-files.js"; +import { skillPackageStore } from "./skill-package-store.js"; import { getRunLogStore } from "./run-log-store.js"; // This is a reviewed catalog of readers, not a proxy for existing GET routes. @@ -281,7 +282,8 @@ export async function readCustomerSuccessResource( assertWorkspaceFilePathAllowed(relative); const versionId = q.versionId ?? skill.currentVersionId; // Installed version snapshots are authoritative. Reading must not invoke - // inventory reconciliation, checkout, fetching, adoption or materialization. + // inventory reconciliation or Git recovery. Stored packages are verified in + // an isolated scratch directory, outside the agent preparation lock. if (versionId) { const version = await ownRow( tx, @@ -290,18 +292,15 @@ export async function readCustomerSuccessResource( String(versionId), ); if (version.companySkillId !== skill.id) throw notFound("Skill version not found"); - const files = version.fileInventory as Array<{ - path: string; - content?: string; - encoding?: string; - }>; - const file = files.find((f) => f.path === relative); + const file = await skillPackageStore(tx, storage, db).readVersionFile( + version as typeof schema.companySkillVersions.$inferSelect, relative, false); if (file?.content !== undefined) { const bytes = Buffer.from(file.content, file.encoding === "base64" ? "base64" : "utf8"); if (bytes.length > MAX_INSPECTION_BYTES) throw unprocessable("Skill snapshot exceeds inspection limit"); return binary(bytes, bytes.length); } + return unavailable("skill_file_not_snapshotted"); } if (relative === "SKILL.md") return binary( diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 0e8748ad226..67cfa6173df 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -99,6 +99,8 @@ export const DELIVERY_QUEUES = { dotEvents: "dot-events", grokDispatch: "grok-dispatch", skillSourceSync: "skill-source-sync", + skillPackages: "skill-packages", + runtimeAssetRetention: "runtime-asset-retention", fastResponse: "fast-responses", feedback: "feedback-exports", chatCompletion: "chat-completions", diff --git a/server/src/services/file-manifest-cache.ts b/server/src/services/file-manifest-cache.ts new file mode 100644 index 00000000000..97c360701a3 --- /dev/null +++ b/server/src/services/file-manifest-cache.ts @@ -0,0 +1,74 @@ +import { createHash } from "node:crypto"; +import { constants, promises as fs } from "node:fs"; +import path from "node:path"; + +export type FileManifestEntry = { path: string; size: number; sha256: string; executable: boolean; mode?: number }; + +// Cache only identities/digests, not file contents. ctime prevents an ordinary +// writer from hiding a modification by restoring mtime. Every use still checks +// paths, file types, count and mode; changed files are hashed again. +const verifiedFiles = new Map(); +const MAX_VERIFIED_FILES = 128_000; +let activeFileIo = 0; +const waitingFileIo: Array<() => void> = []; +async function withFileIo(action: () => Promise): Promise { + if (activeFileIo >= 24) await new Promise(resolve => waitingFileIo.push(resolve)); + else activeFileIo++; + try { return await action(); } + finally { const next = waitingFileIo.shift(); if (next) next(); else activeFileIo--; } +} +/** Rename preserves file identities; carry verification forward to avoid + * re-reading every byte on the first warm turn after a cold materialization. */ +export function relocateFileManifestVerification(from: string, to: string, files: readonly FileManifestEntry[]) { + for (const file of files) { + const source = path.join(from, file.path); + const verified = verifiedFiles.get(source); + verifiedFiles.delete(source); + if (verified) verifiedFiles.set(path.join(to, file.path), verified); + } +} +export async function verifyFileManifest(directory: string, files: readonly FileManifestEntry[]): Promise { + const expected = new Map(files.map(file => [file.path, file])); + const seen = new Set(); + async function walk(relative: string): Promise { + const root = path.join(directory, relative); + const rootStat = await fs.lstat(root); + if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) return false; + const entries = await fs.readdir(root, { withFileTypes: true }); + for (let offset = 0; offset < entries.length; offset += 24) { + const outcomes = await Promise.all(entries.slice(offset, offset + 24).map(async entry => { + const name = relative ? `${relative}/${entry.name}` : entry.name; + if (entry.isDirectory()) return walk(name); + const file = expected.get(name); + if (!file || !entry.isFile()) return false; + seen.add(name); + return withFileIo(async () => { + const target = path.join(directory, name); + const stat = await fs.lstat(target, { bigint: true }); + if (!stat.isFile() || stat.size !== BigInt(file.size) || Boolean(stat.mode & 0o111n) !== file.executable + || (file.mode !== undefined && Number(stat.mode & 0o555n) !== file.mode)) return false; + const signature = `${stat.dev}:${stat.ino}:${stat.size}:${stat.mtimeNs}:${stat.ctimeNs}:${stat.mode}`; + const previous = verifiedFiles.get(target); + if (previous?.signature === signature && previous.digest === file.sha256) return true; + const handle = await fs.open(target, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const before = await handle.stat({ bigint: true }); + if (before.ino !== stat.ino || before.ctimeNs !== stat.ctimeNs) return false; + const hash = createHash("sha256"); + for await (const chunk of handle.createReadStream({ autoClose: false })) hash.update(chunk); + if (hash.digest("hex") !== file.sha256) return false; + const after = await handle.stat({ bigint: true }); + if (after.ctimeNs !== before.ctimeNs || after.mtimeNs !== before.mtimeNs) return false; + if (verifiedFiles.size >= MAX_VERIFIED_FILES) verifiedFiles.delete(verifiedFiles.keys().next().value!); + verifiedFiles.set(target, { signature, digest: file.sha256 }); + return true; + } finally { await handle.close(); } + }); + })); + if (outcomes.some(result => !result)) return false; + } + return true; + } + try { return await walk("") && seen.size === expected.size; } + catch (error) { if (["ENOENT", "ENOTDIR", "ELOOP"].includes((error as NodeJS.ErrnoException).code ?? "")) return false; throw error; } +} diff --git a/server/src/services/github-skill-source.ts b/server/src/services/github-skill-source.ts index f348affd27e..12c5b99168f 100644 --- a/server/src/services/github-skill-source.ts +++ b/server/src/services/github-skill-source.ts @@ -3,7 +3,7 @@ import { MAX_REPOSITORY_BYTES, MAX_REPOSITORY_FILES, repositoryPackageInspection import type { CompanySkillVersionFileInventoryEntry, SkillSourceCandidate, SkillSourceDiscovery, SkillSourcePreviewRequest, SkillSourceFilePreview, SkillSourceScanUpdate, SkillSourceScanProgress } from '@paperclipai/shared'; import { parseFrontmatterMarkdown, parseGitHubSkillRepositoryUrl } from '@paperclipai/shared'; import { notFound, unprocessable } from '../errors.js'; -import { assertSkillSnapshotPath, snapshotFile, skillFileBytes } from './skill-snapshot.js'; +import { assertSkillSnapshotPath, snapshotFile, skillFileBytes, MAX_SKILL_SNAPSHOT_BYTES, MAX_SKILL_SNAPSHOT_FILES } from './skill-snapshot.js'; import { indexSkillPackagePaths, inspectSkillPackage } from './skill-package-inspection.js'; import { auditSkillSnapshot, classifyInventoryKind } from './company-skills.js'; import { skillRepositoryDependencies } from './skill-repository-dependencies.js'; @@ -20,9 +20,9 @@ export interface GitHubRead { export type DiscoveredSkill = SkillSourceCandidate & { files: CompanySkillVersionFileInventoryEntry[] }; export type ScannedSkillSource = SkillSourceDiscovery & { skills: DiscoveredSkill[]; defaultBranch: string; treeSha?: string; downloadBytes?: number; scanBytes?: number; repositoryFiles?: CompanySkillVersionFileInventoryEntry[]; repositorySubset?: boolean }; const MAX_FILE_BYTES = 1024 * 1024; -const MAX_SCAN_BYTES = 100 * 1024 * 1024; +const MAX_SCAN_BYTES = MAX_SKILL_SNAPSHOT_BYTES; const MAX_SCAN_PACKAGES = 1_000; -const MAX_SCAN_FILES = 10_000; +const MAX_SCAN_FILES = MAX_SKILL_SNAPSHOT_FILES; export function parseSkillRepository(url: string) { const parsed = parseGitHubSkillRepositoryUrl(url); @@ -105,6 +105,12 @@ async function scanRepository(input: SkillScanInput, providerRead: GitHubRead, o } const repositoryPaths = indexSkillPackagePaths(entries.map(entry => entry.path)); const roots = entries.filter(e => e.type === 'blob' && /(^|\/)skill\.md$/i.test(e.path) && ['100644', '100755'].includes(e.mode)); + if (!roots.length) { + return { treeSha: snapshot.treeSha, downloadBytes: snapshot.downloadBytes, scanBytes: 0, + packageMode: input.packageMode, connectionId: providerRead.connectionId ?? null, + repositoryId: String(repo.id), repositoryUrl: canonical.repositoryUrl, fullName: repo.full_name, + trackingRef, commitSha: commit.sha, defaultBranch: repo.default_branch, candidates: [], warnings: [], skills: [] }; + } const rootsByDirectory = new Map(); for (const root of roots) { const dir = path.posix.dirname(root.path); diff --git a/server/src/services/heartbeat/run-preparation.ts b/server/src/services/heartbeat/run-preparation.ts index 7d9b506594b..c251379d9fd 100644 --- a/server/src/services/heartbeat/run-preparation.ts +++ b/server/src/services/heartbeat/run-preparation.ts @@ -1,3 +1,4 @@ +import { skillPackageStore, skillVersionMetadataColumns } from "../skill-package-store.js"; import { skillAccessService } from "../skill-access.js"; import { buildReviewPathRecoveryDisposition } from "../recovery/review-path-recovery.js"; import type { AuthorizationActor } from "../authorization.js"; @@ -2960,7 +2961,7 @@ export function createHeartbeatRunPreparation(db: Db) { inputId: companySkillTestRuns.inputId, skillVersionId: companySkillTestRuns.skillVersionId, outputDocumentKey: companySkillTestRuns.outputDocumentKey, - fileInventory: companySkillVersions.fileInventory, + version: skillVersionMetadataColumns, revisionNumber: companySkillVersions.revisionNumber, label: companySkillVersions.label, }) @@ -2980,8 +2981,9 @@ export function createHeartbeatRunPreparation(db: Db) { ) .then((rows) => rows[0] ?? null); if (!row) return null; - const fileInventory = Array.isArray(row.fileInventory) - ? row.fileInventory.flatMap((entry) => { + const savedFiles = await skillPackageStore(db).versionFiles(row.version); + const fileInventory = Array.isArray(savedFiles) + ? savedFiles.flatMap((entry) => { if (!entry || typeof entry !== "object" || Array.isArray(entry)) return []; const record = entry as unknown as Record; diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index 0e7463b2191..b9ee6483e52 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -213,6 +213,9 @@ const WORK_CHECKS = [ // label requires its completion stamp and a cleared failure reason. `SELECT 1 FROM feedback_exports WHERE (status = 'local_only' OR (status = 'sent' AND exported_at IS NOT NULL AND failure_reason IS NULL)) IS NOT TRUE`, + // Skill package conversion and retention are optional maintenance. Legacy + // contents remain authoritative until verified conversion; retries may wait + // for ingress, including when external package storage is disabled. `SELECT 1 FROM project_skill_sources b JOIN companies c ON c.id = b.company_id WHERE c.status = 'active' AND b.active AND b.source_id IS NULL`, `SELECT 1 FROM company_skill_sources s JOIN companies c ON c.id = s.company_id diff --git a/server/src/services/native-runtime/runtime-asset-retention.test.ts b/server/src/services/native-runtime/runtime-asset-retention.test.ts new file mode 100644 index 00000000000..2d8444c0184 --- /dev/null +++ b/server/src/services/native-runtime/runtime-asset-retention.test.ts @@ -0,0 +1,57 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; +import { materializeAsset } from "./runtime-context.js"; +import { removeRuntimeAssetTree, runtimeAssetRetention, RUNTIME_ASSET_RETENTION_MS } from "./runtime-asset-retention.js"; + +let home: string; +const originalHome = process.env.PAPERCLIP_HOME; +beforeEach(async () => { home = await fs.mkdtemp(path.join(os.tmpdir(), "runtime-asset-gc-")); process.env.PAPERCLIP_HOME = home; }); +afterEach(async () => { + await removeRuntimeAssetTree(home); + if (originalHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = originalHome; + vi.restoreAllMocks(); +}); +function database(rows: unknown[] = []) { + const select = vi.fn(() => ({ from: () => ({ where: () => ({ limit: async () => rows }) }) })); + return { db: { select } as unknown as Db, select }; +} +async function age(digest: string) { + const manifest = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets", "manifests", `${digest}.json`); + const old = new Date(Date.now() - RUNTIME_ASSET_RETENTION_MS - 1_000); + await fs.utimes(manifest, old, old); +} +describe("runtime asset retention", () => { + it("has no database work when empty/fresh, removes expired copies and rebuilds the same digest", async () => { + const { db, select } = database(); + const worker = runtimeAssetRetention(db); + await worker.tick(new AbortController().signal); + expect(worker.nextRunAt()).toBeNull(); + expect(select).not.toHaveBeenCalled(); + const files = [{ path: "SKILL.md", content: Buffer.from("immutable skill"), mode: 0o444 }]; + const asset = await materializeAsset(files); + await worker.tick(new AbortController().signal); + expect(worker.nextRunAt()).toBeGreaterThan(Date.now()); + expect(select).not.toHaveBeenCalled(); + await age(asset.digest); + await worker.tick(new AbortController().signal); + expect(select).toHaveBeenCalledOnce(); + expect(await fs.stat(asset.rootPath).catch(() => null)).toBeNull(); + expect(worker.nextRunAt()).toBeNull(); + await worker.tick(new AbortController().signal); + expect(select).toHaveBeenCalledOnce(); + expect(await materializeAsset(files)).toEqual(asset); + }); + it("keeps files when an unfinished run has no trustworthy input", async () => { + const asset = await materializeAsset([{ path: "SKILL.md", content: Buffer.from("pinned skill"), mode: 0o444 }]); + await age(asset.digest); + const { db } = database([{ id: "run", companyId: "company", agentId: "agent", input: null }]); + const worker = runtimeAssetRetention(db); + await worker.tick(new AbortController().signal); + expect((await fs.stat(asset.rootPath)).isDirectory()).toBe(true); + expect(worker.nextRunAt()).toBeNull(); + }); +}); diff --git a/server/src/services/native-runtime/runtime-asset-retention.ts b/server/src/services/native-runtime/runtime-asset-retention.ts new file mode 100644 index 00000000000..2e74881888b --- /dev/null +++ b/server/src/services/native-runtime/runtime-asset-retention.ts @@ -0,0 +1,145 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { and, eq, sql } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { withDirectoryPublicationLock } from "@paperclipai/adapter-utils/workspace-restore-merge"; +import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; +import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; +import { ensureSkillCacheDirectory, skillRunNeedsFilesSql } from "../skill-version-cache.js"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; + +export const RUNTIME_ASSET_RETENTION_MS = 30 * 24 * 60 * 60_000; +const TOUCH_MS = 60 * 60_000; +const touched = new Map(); +const observers = new Set<() => void>(); +const root = () => path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets"); +function paths(digest: string) { + if (!/^[a-f0-9]{64}$/.test(digest)) throw new Error("Invalid runtime asset digest"); + return { bundle: path.join(root(), "bundles", digest), manifest: path.join(root(), "manifests", `${digest}.json`) }; +} + +/** Persist an hourly access lease, under the same filesystem lock as deletion. + * Warm turns inside that lease add no I/O or database calls. The long grace also + * protects the short interval between preparation and durable run pinning. */ +export async function withRuntimeAssetUse(digest: string, consume: (locked: boolean) => Promise, mutation = false): Promise { + const target = paths(digest); + const last = touched.get(target.manifest); + const fresh = last !== undefined && Date.now() - last < TOUCH_MS; + if (fresh && !mutation) return consume(false); + await ensureSkillCacheDirectory(path.dirname(target.bundle), true); + return withDirectoryPublicationLock(target.bundle, async () => { + const result = await consume(true); + if (!fresh && (await fs.lstat(target.manifest).catch(() => null))?.isFile()) { + const now = Date.now(); + await fs.utimes(target.manifest, new Date(now), new Date(now)); + if (touched.size >= 4096) touched.delete(touched.keys().next().value!); + touched.set(target.manifest, now); + for (const wake of observers) wake(); + } + return result; + }); +} + +async function protectedAssets(db: Db): Promise | null> { + const rows = await db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + input: sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'` }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.runtimeMode, "native"), skillRunNeedsFilesSql())).limit(1001); + if (rows.length > 1000) return null; + const pins = new Set(); + for (const row of rows) { + try { + const input = parseNativeExecutionInput(row.input); + if (input.binding.runId !== row.id || input.binding.companyId !== row.companyId || input.binding.agentId !== row.agentId) return null; + if (!("runtimeContext" in input)) return null; + for (const asset of [input.runtimeContext.instructions.bundle, ...input.runtimeContext.skills.map(skill => skill.bundle)]) pins.add(asset.digest); + } catch { return null; } // An unfinished/legacy input cannot prove absence. + } + return pins; +} +export async function removeRuntimeAssetTree(directory: string) { + const stat = await fs.lstat(directory).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (!stat) return; + if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Unsafe runtime asset directory"); + await fs.chmod(directory, 0o700); + for (const entry of await fs.readdir(directory, { withFileTypes: true })) { + if (entry.isDirectory()) await removeRuntimeAssetTree(path.join(directory, entry.name)); + else if (!entry.isFile()) throw new Error("Unsafe runtime asset entry"); + } + await fs.rm(directory, { recursive: true }); +} + +export function runtimeAssetRetention(db: Db) { + let next: number | null = null; + async function tick(signal: AbortSignal) { + next = null; + const directory = path.join(root(), "manifests"); + try { await ensureSkillCacheDirectory(directory); } + catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } + const candidates: Array<{ digest: string; at: number }> = []; + const seen = new Set(); + const manifests = await fs.readdir(directory, { withFileTypes: true }).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return []; throw error; + }); + for (const entry of manifests) { + signal.throwIfAborted(); + if (!entry.isFile() || !/^[a-f0-9]{64}\.json$/.test(entry.name)) continue; + const stat = await fs.lstat(path.join(directory, entry.name)).catch(() => null); + if (!stat?.isFile()) continue; + seen.add(entry.name.slice(0, -5)); + const at = stat.mtimeMs + RUNTIME_ASSET_RETENTION_MS; + if (at <= Date.now()) candidates.push({ digest: entry.name.slice(0, -5), at }); + else next = Math.min(next ?? Infinity, at); + } + // A crash between bundle rename and manifest publication can leave an + // unreferenced bundle. The same age, lock and run-pin checks apply. + const bundles = path.join(root(), "bundles"); + try { + await ensureSkillCacheDirectory(bundles); + for (const entry of await fs.readdir(bundles, { withFileTypes: true })) { + signal.throwIfAborted(); + if (!entry.isDirectory() || !/^[a-f0-9]{64}$/.test(entry.name) || seen.has(entry.name)) continue; + const stat = await fs.lstat(path.join(bundles, entry.name)).catch(() => null); + if (!stat?.isDirectory()) continue; + const at = stat.mtimeMs + RUNTIME_ASSET_RETENTION_MS; + if (at <= Date.now()) candidates.push({ digest: entry.name, at }); + else next = Math.min(next ?? Infinity, at); + } + } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } + if (!candidates.length) return; // No SQL merely to discover idle history. + const pins = await protectedAssets(db); + if (!pins) return; // Reconsider on the next asset use or startup, without polling. + const eligible = candidates.filter(entry => !pins.has(entry.digest)); + for (const { digest } of eligible.slice(0, 32)) { + signal.throwIfAborted(); + const target = paths(digest); + await ensureSkillCacheDirectory(path.dirname(target.bundle), true); + await withDirectoryPublicationLock(target.bundle, async () => { + const stat = await fs.lstat(target.manifest).catch(() => null) ?? await fs.lstat(target.bundle).catch(() => null); + if (!stat || (!stat.isFile() && !stat.isDirectory()) || stat.isSymbolicLink()) return; + const at = stat.mtimeMs + RUNTIME_ASSET_RETENTION_MS; + if (at > Date.now()) { next = Math.min(next ?? Infinity, at); return; } + await removeRuntimeAssetTree(target.bundle); + await fs.rm(target.manifest, { force: true }); + touched.delete(target.manifest); + }); + } + if (eligible.length > 32) next = Math.min(next ?? Infinity, Date.now() + 1_000); + } + function start(coordinator: Pick, "register">) { + let requested = 0, completed = -1; + const worker = coordinator.register(DELIVERY_QUEUES.runtimeAssetRetention, { + retryMs: 5_000, pausePassiveMaintenance: true, + async run(signal) { const generation = requested; await tick(signal); completed = generation; }, + hasPending: async () => false, + nextRunAt: () => requested !== completed ? Date.now() + 5_000 : next, + }); + const wake = () => { requested++; worker.wake(); }; + observers.add(wake); + return { ...worker, async stop() { observers.delete(wake); await worker.stop(); } }; + } + return { tick, start, nextRunAt: () => next }; +} diff --git a/server/src/services/native-runtime/runtime-context.test.ts b/server/src/services/native-runtime/runtime-context.test.ts index 5af85205e21..e6d1fbd9539 100644 --- a/server/src/services/native-runtime/runtime-context.test.ts +++ b/server/src/services/native-runtime/runtime-context.test.ts @@ -1,3 +1,4 @@ +import fs from "node:fs/promises"; import { mkdtemp, mkdir, readFile, readdir, chmod, lstat, rm, stat, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -290,7 +291,11 @@ describe("buildNativeRuntimeContext", () => { }; const context = await buildNativeRuntimeContext(input); + const reads = vi.spyOn(fs, "readFile"); const repeated = await buildNativeRuntimeContext(input); + expect(reads.mock.calls.some(([file]) => String(file).startsWith(selectedRoot) + || String(file).startsWith(context.skills[0]!.bundle.rootPath))).toBe(false); + reads.mockRestore(); expect(serviceMocks.exportFiles).toHaveBeenCalledWith(input.agent, { rejectSymlinks: true }); expect(serviceMocks.getEffectiveToolAssignmentsForAgent).toHaveBeenCalledWith("company-1", "agent-1"); @@ -322,6 +327,19 @@ describe("buildNativeRuntimeContext", () => { expect(repeated.aggregateDigest).toBe(context.aggregateDigest); expect(repeated.instructions.bundle.rootPath).toBe(context.instructions.bundle.rootPath); expect(repeated.skills[0]!.bundle.rootPath).toBe(context.skills[0]!.bundle.rootPath); + await writeFile(path.join(selectedRoot, "references/checklist.md"), "- Verify edits\n"); + const edited = await buildNativeRuntimeContext(input); + expect(edited.skills[0]!.bundle.digest).not.toBe(context.skills[0]!.bundle.digest); + const corrupted = path.join(edited.skills[0]!.bundle.rootPath, "SKILL.md"); + await chmod(corrupted, 0o644); + await writeFile(corrupted, "Corrupted saved asset"); + const repaired = await buildNativeRuntimeContext(input); + expect(repaired.skills[0]!.bundle.digest).toBe(edited.skills[0]!.bundle.digest); + expect(await readFile(corrupted, "utf8")).toContain("Use the checklist"); + await chmod(path.dirname(corrupted), 0o755); + await fs.rm(corrupted); + expect((await buildNativeRuntimeContext(input)).skills[0]!.bundle.digest).toBe(edited.skills[0]!.bundle.digest); + expect(await readFile(corrupted, "utf8")).toContain("Use the checklist"); }); it("does not materialize assigned skills or external MCP in preconfigured mode", async () => { diff --git a/server/src/services/native-runtime/runtime-context.ts b/server/src/services/native-runtime/runtime-context.ts index ee457686486..d0a96fed9de 100644 --- a/server/src/services/native-runtime/runtime-context.ts +++ b/server/src/services/native-runtime/runtime-context.ts @@ -1,3 +1,6 @@ +import { withRuntimeAssetUse, removeRuntimeAssetTree } from "./runtime-asset-retention.js"; +import { cachedSkillAsset, rememberSkillAsset } from "./skill-asset-cache.js"; +import { verifyFileManifest, type FileManifestEntry } from "../file-manifest-cache.js"; import { measureRunPhase } from "../execution-preparation-timing.js"; import { spekoToolsForSession } from "../voice/speko-agent-tools.js"; import { githubBotConnectionIdsForRun } from "../chat-github-tools.js"; @@ -83,14 +86,9 @@ async function verifyMaterializedAsset( rootPath: string, manifestFiles: Array<{ path: string; sha256: string; mode: number; size: number }>, ): Promise { - const actual = await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(rootPath)); - const actualByPath = new Map(actual.map((file) => [file.path, file])); - if (actualByPath.size !== manifestFiles.length) throw new Error("runtime context asset file count mismatch"); - for (const expected of manifestFiles) { - const file = actualByPath.get(expected.path); - if (!file || file.content.byteLength !== expected.size || sha256(file.content) !== expected.sha256 || (file.mode & 0o555) !== expected.mode) { - throw new Error(`runtime context asset digest mismatch: ${expected.path}`); - } + const expected = manifestFiles.map(file => ({ ...file, executable: Boolean(file.mode & 0o111) })); + if (!await measureRunPhase("runtime_context.asset_read", () => verifyFileManifest(rootPath, expected))) { + throw new Error("runtime context asset digest mismatch"); } } @@ -105,55 +103,62 @@ export async function materializeAsset(files: AssetFile[]): Promise fs.readFile(manifestPath, "utf8").catch(() => null)); - if (existing !== null) { - if (sha256(existing) !== manifestDigest || !(await fs.stat(rootPath).catch(() => null))?.isDirectory()) throw new Error(`runtime context asset verification failed: ${assetDigest}`); - await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes }); - return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; - } - await measureRunPhase("runtime_context.asset_materialize", async () => { - const stagingRoot = path.join(assetsRoot, ".staging", randomUUID()); - await fs.mkdir(stagingRoot, { recursive: true, mode: 0o700 }); - try { - for (const file of sorted) { - const target = path.join(stagingRoot, file.path); - await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); - await fs.writeFile(target, file.content, { mode: file.mode & 0o555 }); - await fs.chmod(target, file.mode & 0o555); - } - await fs.mkdir(path.dirname(rootPath), { recursive: true, mode: 0o700 }); - let publishedRoot = false; - await fs.rename(stagingRoot, rootPath).then(() => { - publishedRoot = true; - }).catch(async (error: NodeJS.ErrnoException) => { - // Concurrent publishers can surface EACCES on macOS when the winning - // destination has already been made read-only. Treat that as a race only - // when the content-addressed destination now exists; a genuine permission - // failure with no published destination must still fail loudly. - const publishedByPeer = (await fs.stat(rootPath).catch(() => null))?.isDirectory() === true; - if (error.code !== "EEXIST" && error.code !== "ENOTEMPTY" && !publishedByPeer) throw error; - await fs.rm(stagingRoot, { recursive: true, force: true }); - }); - // Publish the writable staging directory first. Renaming a read-only - // directory can fail with EACCES on macOS even when both parents are owned - // and writable. Files are already immutable; directories become immutable - // immediately after this process wins publication. - if (publishedRoot) await makeDirectoriesReadOnly(rootPath); - await fs.mkdir(path.dirname(manifestPath), { recursive: true, mode: 0o700 }); - await fs.writeFile(manifestPath, manifestText, { flag: "wx", mode: 0o400 }).catch(async (error: NodeJS.ErrnoException) => { - if (error.code !== "EEXIST") throw error; - if (sha256(await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8"))) !== manifestDigest) throw new Error(`runtime context manifest race mismatch: ${assetDigest}`); - }); - await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes }); - } catch (error) { - await fs.rm(stagingRoot, { recursive: true, force: true }).catch(() => undefined); - throw error; + return withRuntimeAssetUse(assetDigest, async function publishOrReuse(locked): Promise { + const assetsRoot = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets"); + const rootPath = path.join(assetsRoot, "bundles", assetDigest); + const manifestPath = path.join(assetsRoot, "manifests", `${assetDigest}.json`); + const existing = await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8").catch(() => null)); + const rootExists = (await fs.stat(rootPath).catch(() => null))?.isDirectory(); + if (existing !== null && rootExists && sha256(existing) === manifestDigest + && await verifyFileManifest(rootPath, manifestFiles.map(file => ({ ...file, executable: Boolean(file.mode & 0o111) })))) { + return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; } - }, { fileCount: manifestFiles.length, bytes: totalBytes }); - return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; + if (!locked) return withRuntimeAssetUse(assetDigest, publishOrReuse, true); + // The caller supplied the exact authoritative bytes for this digest. A missing + // or damaged disposable copy can be rebuilt under the publication/GC lock. + if (rootExists) await removeRuntimeAssetTree(rootPath); + await fs.rm(manifestPath, { force: true }); + await measureRunPhase("runtime_context.asset_materialize", async () => { + const stagingRoot = path.join(assetsRoot, ".staging", randomUUID()); + await fs.mkdir(stagingRoot, { recursive: true, mode: 0o700 }); + try { + for (const file of sorted) { + const target = path.join(stagingRoot, file.path); + await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); + await fs.writeFile(target, file.content, { mode: file.mode & 0o555 }); + await fs.chmod(target, file.mode & 0o555); + } + await fs.mkdir(path.dirname(rootPath), { recursive: true, mode: 0o700 }); + let publishedRoot = false; + await fs.rename(stagingRoot, rootPath).then(() => { + publishedRoot = true; + }).catch(async (error: NodeJS.ErrnoException) => { + // Concurrent publishers can surface EACCES on macOS when the winning + // destination has already been made read-only. Treat that as a race only + // when the content-addressed destination now exists; a genuine permission + // failure with no published destination must still fail loudly. + const publishedByPeer = (await fs.stat(rootPath).catch(() => null))?.isDirectory() === true; + if (error.code !== "EEXIST" && error.code !== "ENOTEMPTY" && !publishedByPeer) throw error; + await fs.rm(stagingRoot, { recursive: true, force: true }); + }); + // Publish the writable staging directory first. Renaming a read-only + // directory can fail with EACCES on macOS even when both parents are owned + // and writable. Files are already immutable; directories become immutable + // immediately after this process wins publication. + if (publishedRoot) await makeDirectoriesReadOnly(rootPath); + await fs.mkdir(path.dirname(manifestPath), { recursive: true, mode: 0o700 }); + await fs.writeFile(manifestPath, manifestText, { flag: "wx", mode: 0o400 }).catch(async (error: NodeJS.ErrnoException) => { + if (error.code !== "EEXIST") throw error; + if (sha256(await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8"))) !== manifestDigest) throw new Error(`runtime context manifest race mismatch: ${assetDigest}`); + }); + await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes }); + } catch (error) { + await fs.rm(stagingRoot, { recursive: true, force: true }).catch(() => undefined); + throw error; + } + }, { fileCount: manifestFiles.length, bytes: totalBytes }); + return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; + }); } async function materializeInstructionBundle(db: Db, agent: RuntimeAgent, agentFiles = false) { @@ -180,7 +185,7 @@ async function materializeInstructionBundle(db: Db, agent: RuntimeAgent, agentFi return { entryPath, bundle: await materializeAsset(files) }; } -async function materializeSelectedSkills(runtimeConfig: Record, entries: PaperclipSkillEntry[], omitLegacy: boolean) { +export async function materializeSelectedSkills(runtimeConfig: Record, entries: PaperclipSkillEntry[], omitLegacy: boolean) { const desiredKeys = resolvePaperclipDesiredSkillNames(runtimeConfig, entries).filter( (key) => !omitLegacy || key !== PAPERCLIP_OPERATIONAL_SKILL_KEY, ); @@ -189,7 +194,17 @@ async function materializeSelectedSkills(runtimeConfig: Record, const entry = byKey.get(key); if (!entry) throw new Error(`assigned runtime skill is missing from the company library: ${key}`); if (entry.sourceStatus === "missing") throw new Error(entry.missingDetail ?? `assigned runtime skill source is missing: ${key}`); - return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle: await materializeAsset(await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(entry.source))) }; + let bundle = await measureRunPhase("runtime_context.asset_verify", () => cachedSkillAsset(entry.source)); + if (!bundle) { + const files = await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(entry.source)); + bundle = await materializeAsset(files); + const manifest: FileManifestEntry[] = files.map(file => ({ path: file.path, size: file.content.length, + sha256: sha256(file.content), mode: file.mode & 0o555, executable: Boolean(file.mode & 0o111) })); + // Populate the stat/digest cache against these exact source bytes before + // remembering the reference. A concurrent source edit cannot seed it. + if (await verifyFileManifest(entry.source, manifest)) rememberSkillAsset(entry.source, manifest, bundle); + } + return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle }; })); } diff --git a/server/src/services/native-runtime/skill-asset-cache.ts b/server/src/services/native-runtime/skill-asset-cache.ts new file mode 100644 index 00000000000..1a85783db5f --- /dev/null +++ b/server/src/services/native-runtime/skill-asset-cache.ts @@ -0,0 +1,46 @@ +import { withRuntimeAssetUse } from "./runtime-asset-retention.js"; +import fs from "node:fs/promises"; +import path from "node:path"; +import { createHash } from "node:crypto"; +import type { NativeRuntimeAssetReference } from "../../vendor/paperclip-runner/index.js"; +import { verifyFileManifest, type FileManifestEntry } from "../file-manifest-cache.js"; +import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; + +type Cached = { manifest: FileManifestEntry[]; bundle: NativeRuntimeAssetReference }; +const entries = new Map(); +let fileCount = 0; +const MAX_FILES = 32_000; +const MAX_ENTRIES = 128; +const cacheKey = (source: string) => `${resolvePaperclipInstanceRoot()}:${source}`; + +/** The caller supplies freshly authorized runtime entries on every turn. This + * cache retains no authority or file bodies, and verifies source and destination + * identities before reusing the content-addressed native asset reference. */ +export async function cachedSkillAsset(source: string): Promise { + const cached = entries.get(cacheKey(source)); + if (!cached || !await verifyFileManifest(source, cached.manifest)) return null; + return withRuntimeAssetUse(cached.bundle.digest, async () => { + const manifestPath = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets", "manifests", `${cached.bundle.digest}.json`); + const manifest = await fs.readFile(manifestPath).catch(() => null); + if (!manifest || !(await fs.lstat(cached.bundle.rootPath).catch(() => null))?.isDirectory()) return null; + if (createHash("sha256").update(manifest).digest("hex") !== cached.bundle.manifestDigest + || !await verifyFileManifest(cached.bundle.rootPath, cached.manifest)) { + return null; + } + return cached.bundle; + }); +} + +export function rememberSkillAsset(source: string, manifest: FileManifestEntry[], bundle: NativeRuntimeAssetReference) { + if (manifest.length > MAX_FILES) return; + const key = cacheKey(source); + const previous = entries.get(key); + if (previous) { fileCount -= previous.manifest.length; entries.delete(key); } + while (entries.size && (fileCount + manifest.length > MAX_FILES || entries.size >= MAX_ENTRIES)) { + const first = entries.keys().next().value!; + fileCount -= entries.get(first)!.manifest.length; + entries.delete(first); + } + entries.set(key, { manifest, bundle }); + fileCount += manifest.length; +} diff --git a/server/src/services/skill-local-import-boundary.test.ts b/server/src/services/skill-local-import-boundary.test.ts new file mode 100644 index 00000000000..e049452fc07 --- /dev/null +++ b/server/src/services/skill-local-import-boundary.test.ts @@ -0,0 +1,42 @@ +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { assertLocalImportOutsideSkillPackageCache } from "./skill-local-import-boundary.js"; + +describe("local import package-cache boundary", () => { + let root: string; + let managed: string; + let cache: string; + beforeEach(async () => { + root = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "skill-import-boundary-"))); + managed = path.join(root, "skills", "company"); + cache = path.join(managed, "__packages__"); + await fs.mkdir(path.join(cache, "snapshot", "private"), { recursive: true }); + await fs.writeFile(path.join(cache, "snapshot", "private", "SKILL.md"), "Private instructions"); + }); + afterEach(() => fs.rm(root, { recursive: true, force: true })); + + it.each(["__packages__", "__packages__/snapshot", "__packages__/snapshot/private", + "__packages__/snapshot/private/SKILL.md", ".", ".."])('rejects cache or ancestor source "%s"', async source => { + await expect(assertLocalImportOutsideSkillPackageCache(await fs.realpath(path.join(managed, source)), managed)) + .rejects.toMatchObject({ status: 422, details: { code: "skill_package_cache_import_denied" } }); + }); + + it("rejects a symlink alias of the cache", async () => { + const alias = path.join(managed, "alias"); + await fs.symlink(path.join(cache, "snapshot"), alias, "dir"); + await expect(assertLocalImportOutsideSkillPackageCache(await fs.realpath(alias), managed)).rejects.toThrow("package caches"); + }); + + it("rejects recursive ancestors before a concurrent fill can create the cache", async () => { + await fs.rm(cache, { recursive: true }); + await expect(assertLocalImportOutsideSkillPackageCache(managed, managed)).rejects.toThrow("package caches"); + }); + + it.each(["ordinary-skill", "__packages__-backup"])("allows unrelated sibling %s", async name => { + const source = path.join(managed, name); + await fs.mkdir(source); + await expect(assertLocalImportOutsideSkillPackageCache(await fs.realpath(source), managed)).resolves.toBeUndefined(); + }); +}); diff --git a/server/src/services/skill-local-import-boundary.ts b/server/src/services/skill-local-import-boundary.ts new file mode 100644 index 00000000000..ab5ad833486 --- /dev/null +++ b/server/src/services/skill-local-import-boundary.ts @@ -0,0 +1,22 @@ +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { unprocessable } from "../errors.js"; + +function contains(root: string, candidate: string) { + const relative = path.relative(root, candidate); + return relative === "" || (!path.isAbsolute(relative) && relative !== ".." && !relative.startsWith(`..${path.sep}`)); +} + +/** Package caches have package-level provenance, not local-import authority. */ +export async function assertLocalImportOutsideSkillPackageCache(sourceRealPath: string, managedSkillsRoot: string) { + const managedRoot = await fs.realpath(managedSkillsRoot).catch(() => null); + if (!managedRoot) return; + const reservedRoot = path.join(managedRoot, "__packages__"); + const canonicalRoot = await fs.realpath(reservedRoot).catch(() => reservedRoot); + // Also reject ancestors before a cache exists: a concurrent fill must not + // become visible to a recursive import after its boundary check. + if ([reservedRoot, canonicalRoot].some(root => contains(root, sourceRealPath) || contains(sourceRealPath, root))) { + throw unprocessable("Skill package caches cannot be imported as local sources. Fork the installed skill to retain its access restrictions.", + { code: "skill_package_cache_import_denied" }); + } +} diff --git a/server/src/services/skill-package-cache.ts b/server/src/services/skill-package-cache.ts new file mode 100644 index 00000000000..da6d0dbc4a6 --- /dev/null +++ b/server/src/services/skill-package-cache.ts @@ -0,0 +1,78 @@ +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { and, eq, inArray } from "drizzle-orm"; +import { companySkillRepositorySnapshots as snapshots, type Db } from "@paperclipai/db"; +import { ensureSkillCacheDirectory } from "./skill-version-cache.js"; + +const MAX_BYTES = 512 * 1024 * 1024; +const MAX_PACKAGES = 64; +const used = new Map(); +export function touchSkillPackage(directory: string) { + if (used.size >= 4096) used.delete(used.keys().next().value!); + used.set(directory, Date.now()); +} + +/** Called only after a cold fill, under the company cache lock. Never on a warm + * check. Version projections and run pins have their own existing retention. */ +export async function trimSkillPackageCache(db: Db, companyId: string, directory: string, + verifyDurable: (id: string, signal: AbortSignal) => Promise, + removeVerified: (action: () => Promise) => Promise = action => action(), cancellation?: AbortSignal) { + cancellation?.throwIfAborted(); + const parent = path.dirname(directory); + await ensureSkillCacheDirectory(parent); + const contents = await fs.readdir(parent, { withFileTypes: true }); + for (const entry of contents) { + if (!/^\.(package|upload|verify)-[a-zA-Z0-9]{6}(\.tar\.gz)?$/.test(entry.name)) continue; + const target = path.join(parent, entry.name); + // Every I/O attempt has a three-minute deadline; leave a much longer crash + // grace so housekeeping cannot race a live publisher in another process. + const stat = await fs.lstat(target).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; + return null; + }); + if (stat && Date.now() - stat.mtimeMs > 60 * 60_000) await fs.rm(target, { recursive: true, force: true }); + } + const names = contents.filter(entry => entry.isDirectory() && /^[a-f0-9-]{36}$/i.test(entry.name)).map(entry => entry.name); + if (!names.length) return; + const rows = await db.select({ id: snapshots.id, storage: snapshots.storage }).from(snapshots) + .where(and(eq(snapshots.companyId, companyId), inArray(snapshots.id, names))); + const byId = new Map(rows.map(row => [row.id, row.storage])); + const candidates = await Promise.all(names.map(async id => { + const location = path.join(parent, id); + const receipt = byId.get(id); + const stat = await fs.lstat(location).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; + return null; + }); + if (!stat) return null; + return { id, location, receipt, lastUsed: used.get(location) ?? stat.mtimeMs, + // Include an upper bound on block rounding and directory/manifest overhead. + bytes: receipt ? receipt.totalBytes + receipt.fileCount * 8192 : 64 * 1024 * 1024 }; + })); + const entries = candidates.filter(entry => entry !== null); + let bytes = entries.reduce((total, entry) => total + entry.bytes, 0); + let count = entries.length; + const timeout = AbortSignal.timeout(10_000); + const signal = cancellation ? AbortSignal.any([cancellation, timeout]) : timeout; + let attempted = 0; + for (const entry of entries.sort((a, b) => a.lastUsed - b.lastUsed)) { + if (bytes <= MAX_BYTES && count <= MAX_PACKAGES) break; + if (signal.aborted || attempted >= 2) break; + if (entry.location === directory) continue; + attempted++; + // A storage outage must not discard the last recoverable copy. Keep the + // excess until the next cold fill can confirm a durable copy independently. + if (byId.has(entry.id) && !await verifyDurable(entry.id, signal)) break; + let removed = false; + await removeVerified(async () => { + if ((used.get(entry.location) ?? entry.lastUsed) > entry.lastUsed) return; + await fs.rm(entry.location, { recursive: true, force: true }); + await fs.rm(path.join(parent, `${entry.id}.json`), { force: true }); + used.delete(entry.location); + removed = true; + }); + if (!removed) continue; + bytes -= entry.bytes; + count--; + } +} diff --git a/server/src/services/skill-package-files.ts b/server/src/services/skill-package-files.ts new file mode 100644 index 00000000000..b93fb176be6 --- /dev/null +++ b/server/src/services/skill-package-files.ts @@ -0,0 +1,146 @@ +import { createHash } from "node:crypto"; +import { constants, createReadStream, createWriteStream, promises as fs } from "node:fs"; +import path from "node:path"; +import { Transform } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import { createGzip, createGunzip } from "node:zlib"; +import { extract, pack } from "tar-stream"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { REPOSITORY_CONTENT_DIR } from "./skill-repository-package.js"; +import { assertSkillSnapshotPath, skillFileBytes, skillSnapshotHash, MAX_SKILL_SNAPSHOT_BYTES, MAX_SKILL_SNAPSHOT_FILES } from "./skill-snapshot.js"; + +export const MAX_SKILL_PACKAGE_BYTES = MAX_SKILL_SNAPSHOT_BYTES; +export const MAX_SKILL_PACKAGE_FILES = MAX_SKILL_SNAPSHOT_FILES; +export const MAX_SKILL_MANIFEST_BYTES = 48 * 1024 * 1024; +export type SkillPackageFile = { + path: string; + kind: CompanySkillVersionFileInventoryEntry["kind"]; + encoding: "utf8" | "base64"; + executable: boolean; + size: number; + sha256: string; +}; +export type SkillPackageManifest = { + format: "paperclip-skill-package/v1"; + contentHash: string; + totalBytes: number; + files: SkillPackageFile[]; +}; +const kinds = new Set(["skill", "markdown", "reference", "script", "asset", "other"]); +export const packageSha256 = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex"); + +export function validateSkillPackageManifest(value: unknown): SkillPackageManifest { + const manifest = value as SkillPackageManifest; + if (!manifest || manifest.format !== "paperclip-skill-package/v1" || !/^[a-f0-9]{64}$/.test(manifest.contentHash) + || !Array.isArray(manifest.files) || !manifest.files.length || manifest.files.length > MAX_SKILL_PACKAGE_FILES) { + throw new Error("Invalid skill package manifest"); + } + const paths = new Set(); + let total = 0; + for (const file of manifest.files) { + assertSkillSnapshotPath(file.path); + // A portable repository fork adds one generated directory to scanner-accepted paths. + const sourcePath = file.path.startsWith(`${REPOSITORY_CONTENT_DIR}/`) ? file.path.slice(REPOSITORY_CONTENT_DIR.length + 1) : file.path; + if (sourcePath.length > 4096 || sourcePath.split("/").length > 64 || paths.has(file.path) + || !kinds.has(file.kind) || !["utf8", "base64"].includes(file.encoding) || typeof file.executable !== "boolean" + || !Number.isSafeInteger(file.size) || file.size < 0 || !/^[a-f0-9]{64}$/.test(file.sha256)) throw new Error("Invalid skill package file"); + paths.add(file.path); + total += file.size; + } + for (const file of manifest.files) { + let parent = path.posix.dirname(file.path); + while (parent !== ".") { + if (paths.has(parent)) throw new Error("Conflicting skill package paths"); + parent = path.posix.dirname(parent); + } + } + if (total !== manifest.totalBytes || total > MAX_SKILL_PACKAGE_BYTES) throw new Error("Skill package exceeds its byte bound"); + return manifest; +} + +export function skillPackageManifest(files: CompanySkillVersionFileInventoryEntry[]): SkillPackageManifest { + return validateSkillPackageManifest({ + format: "paperclip-skill-package/v1", contentHash: skillSnapshotHash(files), + totalBytes: files.reduce((sum, file) => sum + skillFileBytes(file).length, 0), + files: files.map(file => { + const bytes = skillFileBytes(file); + return { path: file.path, kind: file.kind, encoding: file.encoding ?? "utf8", executable: Boolean(file.executable), size: bytes.length, sha256: packageSha256(bytes) }; + }), + }); +} + +/** JSONB reorders object keys. Preserve the archive manifest's v1 wire encoding + * when a recovery recipe is read back from PostgreSQL. Array order is retained. */ +export function skillPackageManifestBytes(manifest: SkillPackageManifest): Buffer { + return Buffer.from(JSON.stringify({ format: manifest.format, contentHash: manifest.contentHash, totalBytes: manifest.totalBytes, + files: manifest.files.map(file => ({ path: file.path, kind: file.kind, encoding: file.encoding, + executable: file.executable, size: file.size, sha256: file.sha256 })) })); +} + +export function boundedHashStream(maxBytes: number) { + const hash = createHash("sha256"); + let bytes = 0; + const stream = new Transform({ transform(chunk: Buffer, _encoding, done) { + bytes += chunk.length; + if (bytes > maxBytes) return done(new Error("Skill package stream exceeds its byte bound")); + hash.update(chunk); + done(null, chunk); + } }); + return { stream, result: () => ({ bytes, sha256: hash.digest("hex") }) }; +} + +export async function writeSkillPackageArchive(filename: string, files: CompanySkillVersionFileInventoryEntry[], signal?: AbortSignal) { + const archive = pack(); + const digest = boundedHashStream(MAX_SKILL_PACKAGE_BYTES + MAX_SKILL_MANIFEST_BYTES); + const writing = pipeline(archive, createGzip({ level: 3 }), digest.stream, createWriteStream(filename, { flags: "wx", mode: 0o600 }), { signal }); + // Attach a handler before producing entries so an early filesystem failure is observed. + void writing.catch(() => {}); + try { + for (const file of [...files].sort((a, b) => a.path.localeCompare(b.path))) { + signal?.throwIfAborted(); + await new Promise((resolve, reject) => archive.entry({ + name: `files/${assertSkillSnapshotPath(file.path)}`, type: "file", mode: file.executable ? 0o555 : 0o444, + mtime: new Date(0), uid: 0, gid: 0, + }, skillFileBytes(file), error => error ? reject(error) : resolve())); + } + archive.finalize(); + await writing; + return digest.result(); + } catch (error) { + archive.destroy(error as Error); + await writing.catch(() => {}); + throw error; + } +} + +export async function extractSkillPackageArchive(filename: string, directory: string, manifest: SkillPackageManifest, signal?: AbortSignal) { + const archive = extract(); + const expected = new Map(manifest.files.map(file => [`files/${file.path}`, file])); + const expanded = boundedHashStream(MAX_SKILL_PACKAGE_BYTES + MAX_SKILL_PACKAGE_FILES * 8192); + const reading = pipeline(createReadStream(filename), createGunzip(), expanded.stream, archive, { signal }); + void reading.catch(() => {}); + try { + for await (const entry of archive) { + signal?.throwIfAborted(); + const file = expected.get(entry.header.name); + if (!file || entry.header.type !== "file" || entry.header.size !== file.size + || entry.header.mode !== (file.executable ? 0o555 : 0o444)) throw new Error("Unexpected skill package archive entry"); + expected.delete(entry.header.name); + const target = path.join(directory, file.path); + await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); + const digest = boundedHashStream(file.size); + const output = await fs.open(target, constants.O_CREAT | constants.O_EXCL | constants.O_WRONLY | constants.O_NOFOLLOW, file.executable ? 0o555 : 0o444); + await pipeline(entry, digest.stream, output.createWriteStream(), { signal }); + const actual = digest.result(); + if (actual.bytes !== file.size || actual.sha256 !== file.sha256) throw new Error("Skill package file checksum mismatch"); + } + await reading; + if (expected.size) throw new Error("Skill package archive is incomplete"); + } catch (error) { + archive.destroy(error as Error); + await reading.catch(() => {}); + throw error; + } +} + +export { verifyFileManifest as verifySkillPackageDirectory, relocateFileManifestVerification as relocateSkillPackageVerification } from "./file-manifest-cache.js"; diff --git a/server/src/services/skill-package-maintenance.ts b/server/src/services/skill-package-maintenance.ts new file mode 100644 index 00000000000..c6b714205ee --- /dev/null +++ b/server/src/services/skill-package-maintenance.ts @@ -0,0 +1,136 @@ +import path from "node:path"; +import { and, asc, eq, gt, isNull, lte, sql } from "drizzle-orm"; +import { companySkillRepositorySnapshots as snapshots, companySkillVersions as versions, + skillPackageDeletions, type Db } from "@paperclipai/db"; +import { getStorageService, type StorageService } from "../storage/index.js"; +import { skillPackageStore, skillVersionFileMap, skillVersionMetadataColumns } from "./skill-package-store.js"; +import { packageSha256 } from "./skill-package-files.js"; +import { skillFileBytes, skillSnapshotHash } from "./skill-snapshot.js"; + +type PageOptions = { id?: string; signal?: AbortSignal; companyId?: string; after?: string; limit?: number; apply?: boolean; clearLegacy?: boolean }; +const pageSize = (options: PageOptions) => Math.min(50, Math.max(1, options.limit ?? 10)); + +/** Bounded maintenance shared by automatic delivery and optional operator diagnostics. */ +export function skillPackageMaintenance(db: Db, storage?: StorageService) { + const store = skillPackageStore(db, storage); + async function migrateSnapshots(options: PageOptions = {}) { + const rows = await db.select({ id: snapshots.id, companyId: snapshots.companyId, contentHash: snapshots.contentHash }) + .from(snapshots).where(and(options.companyId ? eq(snapshots.companyId, options.companyId) : undefined, + options.id ? eq(snapshots.id, options.id) : undefined, + options.after ? gt(snapshots.id, options.after) : undefined, options.clearLegacy ? undefined : isNull(snapshots.storage))) + .orderBy(asc(snapshots.id)).limit(pageSize(options)); + let migrated = 0; + if (options.apply) for (const row of rows) { + // Transfer only one legacy body at a time. Never trust a mutable upstream. + const [legacy] = await db.select({ files: snapshots.fileInventory }).from(snapshots).where(eq(snapshots.id, row.id)); + options.signal?.throwIfAborted(); + if (!legacy?.files.length) { + await db.update(snapshots).set({ migrationAt: null }).where(eq(snapshots.id, row.id)); + continue; + } + if (skillSnapshotHash(legacy.files) !== row.contentHash) throw new Error(`Legacy package hash mismatch: ${row.id}`); + const id = await store.prepare(row.companyId, legacy.files, true, options.signal); + if (id !== row.id || !await store.verifyDurable(row.companyId, id, options.signal)) throw new Error(`Package verification failed: ${row.id}`); + // An ambiguous result is safe to retry: storage is already published, IDs + // are unchanged, and the old bytes survive until this final update. + if (options.clearLegacy) await db.update(snapshots).set({ fileInventory: [], migrationAt: null }).where(and(eq(snapshots.id, id), eq(snapshots.contentHash, row.contentHash), sql`${snapshots.storage} is not null`)); + migrated++; + } + return { inspected: rows.length, migrated, after: rows.at(-1)?.id ?? null }; + } + + async function migrateVersions(options: PageOptions = {}) { + const rows = await db.select(skillVersionMetadataColumns).from(versions).where(and( + options.companyId ? eq(versions.companyId, options.companyId) : undefined, + options.id ? eq(versions.id, options.id) : undefined, + options.after ? gt(versions.id, options.after) : undefined, options.clearLegacy ? undefined : isNull(versions.packageSnapshotId), isNull(versions.releaseId))) + .orderBy(asc(versions.id)).limit(pageSize(options)); + let migrated = 0; + if (options.apply) for (const row of rows) { + const [legacy] = await db.select({ files: versions.fileInventory }).from(versions).where(eq(versions.id, row.id)); + const files = legacy?.files; + options.signal?.throwIfAborted(); + if (!files?.length) { + await db.update(versions).set({ migrationAt: null }).where(eq(versions.id, row.id)); + continue; + } + if (row.repositorySnapshotId) await migrateSnapshots({ id: row.repositorySnapshotId, apply: true, clearLegacy: true, signal: options.signal }); + let packageId = row.packageSnapshotId; + let fileMap = row.packageFileMap; + if (packageId && skillSnapshotHash(await store.versionFiles(row, true, options.signal)) !== skillSnapshotHash(files)) throw new Error(`Legacy version differs from its package: ${row.id}`); + if (!packageId && row.repositorySnapshotId && row.repositorySkillPath) { + const manifest = await store.manifest(row.companyId, row.repositorySnapshotId, options.signal); + const byPath = new Map(manifest.files.map(file => [file.path, file])); + const mappings = files.map(file => ({ path: file.path, kind: file.kind, encoding: file.encoding ?? "utf8" as const, + sourcePath: file.path === "SKILL.md" ? row.repositorySkillPath! : path.posix.join(path.posix.dirname(row.repositorySkillPath!), file.path) })); + if (files.every((file, index) => { + const source = byPath.get(mappings[index]!.sourcePath); + return source?.sha256 === packageSha256(skillFileBytes(file)) && source.executable === Boolean(file.executable); + })) { packageId = row.repositorySnapshotId; fileMap = mappings; } + } + packageId ??= await store.prepare(row.companyId, files, true, options.signal); + const legacyMetadata = new Map(files.map(file => [file.path, file])); + fileMap = fileMap ? fileMap.map(file => ({ ...file, kind: legacyMetadata.get(file.path)?.kind ?? file.kind, + encoding: legacyMetadata.get(file.path)?.encoding ?? "utf8" })) : skillVersionFileMap(files, await store.manifest(row.companyId, packageId, options.signal)); + const receipt = await store.descriptor(row.companyId, packageId); + if (!receipt.storage || !await store.verifyDurable(row.companyId, packageId, options.signal)) throw new Error(`Package must be migrated and verified first: ${packageId}`); + await db.update(versions).set({ packageSnapshotId: packageId, packageFileMap: fileMap, ...(options.clearLegacy ? { fileInventory: [], migrationAt: null } : {}) }) + .where(and(eq(versions.companyId, row.companyId), eq(versions.id, row.id), row.packageSnapshotId ? eq(versions.packageSnapshotId, row.packageSnapshotId) : isNull(versions.packageSnapshotId))); + migrated++; + } + return { inspected: rows.length, migrated, after: rows.at(-1)?.id ?? null }; + } + + async function collectOrphans(options: PageOptions = {}) { + const unused = sql`not exists (select 1 from ${versions} where ${versions.repositorySnapshotId} = ${snapshots.id} or ${versions.packageSnapshotId} = ${snapshots.id})`; + const expired = sql`(${snapshots.upload} is null or (${snapshots.upload}->>'expiresAt')::timestamptz < now())`; + const rows = await db.select({ id: snapshots.id }).from(snapshots).where(and( + options.companyId ? eq(snapshots.companyId, options.companyId) : undefined, + options.id ? eq(snapshots.id, options.id) : undefined, + options.after ? gt(snapshots.id, options.after) : undefined, + sql`${snapshots.createdAt} < now() - interval '1 day'`, + expired, unused, + )).orderBy(asc(snapshots.id)).limit(pageSize(options)); + let removed = 0; + if (options.apply) for (const row of rows) { + // FK locking wins against a concurrent publication. The trigger persists + // deletion receipts in the same transaction, including company cascades. + const deleted = await db.delete(snapshots).where(and(eq(snapshots.id, row.id), expired, unused)).returning({ id: snapshots.id }); + removed += deleted.length; + } + return { inspected: rows.length, removed, after: rows.at(-1)?.id ?? null }; + } + + async function deleteObjects(options: PageOptions = {}) { + const backend = storage ?? getStorageService(); + const rows = await db.select().from(skillPackageDeletions).where(and( + options.companyId ? eq(skillPackageDeletions.companyId, options.companyId) : undefined, + options.id ? eq(skillPackageDeletions.id, options.id) : undefined, + options.after ? gt(skillPackageDeletions.id, options.after) : undefined, + lte(skillPackageDeletions.deleteAfter, new Date()), eq(skillPackageDeletions.provider, backend.provider), + )).orderBy(asc(skillPackageDeletions.id)).limit(pageSize(options)); + let removed = 0; + if (options.apply) for (const row of rows) { + options.signal?.throwIfAborted(); + const [referenced] = await db.select({ id: snapshots.id }).from(snapshots).where(and(eq(snapshots.companyId, row.companyId), + sql`(${snapshots.storage}->>'archiveKey' = ${row.archiveKey} or ${snapshots.upload}->>'archiveKey' = ${row.archiveKey})`)).limit(1); + if (referenced) { + // Live ownership supersedes this receipt. A future replacement/deletion + // enqueues fresh work in the same transaction as the ownership change. + await db.delete(skillPackageDeletions).where(eq(skillPackageDeletions.id, row.id)); + continue; + } + const timeout = AbortSignal.timeout(30_000); + const signal = options.signal ? AbortSignal.any([options.signal, timeout]) : timeout; + for (const key of [row.archiveKey, row.manifestKey]) { + await backend.deleteObject(row.companyId, key, { signal }); + if ((await backend.headObject(row.companyId, key, { signal })).exists) throw new Error(`Storage did not confirm package deletion: ${row.id}`); + } + signal.throwIfAborted(); + await db.delete(skillPackageDeletions).where(eq(skillPackageDeletions.id, row.id)); + removed++; + } + return { inspected: rows.length, removed, after: rows.at(-1)?.id ?? null }; + } + return { migrateSnapshots, migrateVersions, collectOrphans, deleteObjects }; +} diff --git a/server/src/services/skill-package-recovery.test.ts b/server/src/services/skill-package-recovery.test.ts new file mode 100644 index 00000000000..b59a705ca48 --- /dev/null +++ b/server/src/services/skill-package-recovery.test.ts @@ -0,0 +1,63 @@ +import { describe, expect, it, vi } from "vitest"; +import { randomUUID } from "node:crypto"; +import { recoverSkillPackage } from "./skill-package-recovery.js"; +import { skillPackageManifest } from "./skill-package-files.js"; +import type { GitHubRead } from "./github-skill-source.js"; +import type { SkillPackageRestoreRecipe } from "@paperclipai/db"; +import { MAX_REPOSITORY_BYTES, MAX_REPOSITORY_FILES } from "./skill-repository-package.js"; + +function fixture() { + const files = [{ path: "SKILL.md", kind: "skill" as const, content: "---\nname: original\n---\nold commit" }, + { path: "run.sh", kind: "script" as const, content: "echo original", executable: true }]; + const recipe: SkillPackageRestoreRecipe = { version: 1, sourceId: randomUUID(), repositoryId: "123", repositoryUrl: "https://github.com/acme/skills", + commitSha: "a".repeat(40), skillPath: "skills/one/skill.md", manifest: skillPackageManifest(files) }; + const entries = recipe.manifest.files.map((file, i) => ({ path: `skills/one/${i ? "run.sh" : "skill.md"}`, type: "blob", mode: file.executable ? "100755" : "100644", sha: String(i).repeat(40), size: file.size })); + const release = vi.fn(async () => {}); + const readBlob = vi.fn(async (entry: typeof entries[number]) => Buffer.from(files[entries.indexOf(entry)]!.content)); + const read = Object.assign(vi.fn(async () => ({ id: 123 })), { + openSnapshot: vi.fn(async () => ({ commitSha: recipe.commitSha, entries, readBlob, release })), + }) as unknown as GitHubRead; + return { files, recipe, read, entries, readBlob, release }; +} +describe("immutable skill package recovery", () => { + it.each([{ count: 9_991, bytes: 1 }, { count: 65, bytes: 1024 * 1024 }])("recovers a package across Git batch limits ($count files of $bytes bytes)", async ({ count, bytes }) => { + const content = "x".repeat(bytes); + const files = Array.from({ length: count }, (_, index) => ({ path: `file-${index}`, kind: "other" as const, content })); + const recipe = { ...fixture().recipe, skillPath: null, manifest: skillPackageManifest(files) }; + const entries = recipe.manifest.files.map((file, index) => ({ path: file.path, type: "blob", mode: "100644", size: file.size, + sha: index.toString(16).padStart(40, "0") })); + const readBlobs = vi.fn(async (batch: typeof entries, signal?: AbortSignal) => { + signal?.throwIfAborted(); + if (batch.length > MAX_REPOSITORY_FILES || batch.reduce((sum, entry) => sum + entry.size, 0) > MAX_REPOSITORY_BYTES) throw new Error("Invalid repository file batch"); + return new Map(batch.map(entry => [entry.sha, Buffer.from(content)])); + }); + const readBlob = vi.fn(async () => { throw new Error("Unexpected individual Git read"); }); + const release = vi.fn(async () => {}); + const read = Object.assign(vi.fn(async () => ({ id: 123 })), { + openSnapshot: vi.fn(async () => ({ commitSha: recipe.commitSha, entries, readBlobs, readBlob, release })), + }) as unknown as GitHubRead; + const restored = await recoverSkillPackage(recipe, read, new AbortController().signal); + expect(skillPackageManifest(restored).contentHash).toBe(recipe.manifest.contentHash); + expect(readBlobs).toHaveBeenCalledTimes(2); + expect(readBlob).not.toHaveBeenCalled(); + expect(release).toHaveBeenCalledOnce(); + }, 30_000); + it("reads the original manifest at the exact commit, including normalized entrypoint and modes", async () => { + const f = fixture(); + expect(await recoverSkillPackage(f.recipe, f.read, new AbortController().signal)).toEqual(f.files); + expect(f.read.openSnapshot).toHaveBeenCalledWith({ repositoryUrl: f.recipe.repositoryUrl, ref: f.recipe.commitSha, commitSha: f.recipe.commitSha }, expect.anything()); + expect(f.read).toHaveBeenCalledExactlyOnceWith("/repos/acme/skills", expect.any(AbortSignal)); + expect(f.release).toHaveBeenCalledOnce(); + }); + it("rejects reused repository names, wrong commits, changed contents and missing files", async () => { + const f = fixture(); + vi.mocked(f.read).mockResolvedValueOnce({ id: 999 }); + await expect(recoverSkillPackage(f.recipe, f.read, new AbortController().signal)).rejects.toThrow("not the original"); + expect(f.read.openSnapshot).not.toHaveBeenCalled(); + f.readBlob.mockResolvedValueOnce(Buffer.from("wrong")); + await expect(recoverSkillPackage(f.recipe, f.read, new AbortController().signal)).rejects.toThrow("checksum mismatch"); + f.entries.pop(); + await expect(recoverSkillPackage(f.recipe, f.read, new AbortController().signal)).rejects.toThrow("unavailable"); + expect(f.release).toHaveBeenCalledTimes(2); + }); +}); diff --git a/server/src/services/skill-package-recovery.ts b/server/src/services/skill-package-recovery.ts new file mode 100644 index 00000000000..91895a3962d --- /dev/null +++ b/server/src/services/skill-package-recovery.ts @@ -0,0 +1,75 @@ +import path from "node:path"; +import { and, eq } from "drizzle-orm"; +import { companySkillSources, type Db, type SkillPackageRestoreRecipe } from "@paperclipai/db"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { forbidden } from "../errors.js"; +import { assertSkillSnapshotPath } from "./skill-snapshot.js"; +import { packageSha256, skillPackageManifest, validateSkillPackageManifest } from "./skill-package-files.js"; +import type { GitHubRead } from "./github-skill-source.js"; +import { MAX_REPOSITORY_BYTES, MAX_REPOSITORY_FILES } from "./skill-repository-package.js"; + +/** Recheck the source's saved identity and current grant. Never search another + * company's credentials or turn a mutable branch name into a historical commit. */ +export async function skillPackageRecoveryReader(db: Db, companyId: string, recipe: SkillPackageRestoreRecipe) { + const [source] = await db.select().from(companySkillSources).where(and( + eq(companySkillSources.companyId, companyId), eq(companySkillSources.id, recipe.sourceId))); + if (!source || source.removedAt || !source.enabled || source.repositoryId !== recipe.repositoryId + || source.repositoryUrl.toLowerCase() !== recipe.repositoryUrl.toLowerCase()) { + throw forbidden("This skill package is no longer cached. Reconnect its original repository to restore it."); + } + // Deferred import avoids making the package store own source synchronization. + const { skillSourceSetupContext } = await import("./skill-source-sync.js"); + const context = await skillSourceSetupContext(db, source); + await context.authorize("skills.update", { sourceType: "git", sourceLocator: recipe.repositoryUrl }); + return context.read(source.connectionId); +} + +export async function recoverSkillPackage(recipe: SkillPackageRestoreRecipe, read: GitHubRead, signal: AbortSignal): Promise { + if (recipe.version !== 1 || !/^[a-f0-9]{40}$/i.test(recipe.commitSha)) throw new Error("Invalid immutable skill package recipe"); + const manifest = validateSkillPackageManifest(recipe.manifest); + const url = new URL(recipe.repositoryUrl); + if (url.protocol !== "https:" || url.hostname !== "github.com" || url.username || url.password + || !/^\/[^/]+\/[^/]+$/.test(url.pathname) || url.search || url.hash) throw new Error("Invalid skill repository identity"); + const repository = await read(`/repos${url.pathname}`, signal) as { id?: number | string }; + if (String(repository.id) !== recipe.repositoryId) throw new Error("The repository at this URL is not the original skill repository"); + const snapshot = await read.openSnapshot({ repositoryUrl: recipe.repositoryUrl, ref: recipe.commitSha, commitSha: recipe.commitSha }, { signal }); + try { + if (snapshot.commitSha !== recipe.commitSha) throw new Error("Git returned a different skill commit"); + const byPath = new Map(snapshot.entries.map(entry => [entry.path, entry])); + if (recipe.skillPath) assertSkillSnapshotPath(recipe.skillPath); + const entries = manifest.files.map(file => { + const sourcePath = recipe.skillPath + ? file.path === "SKILL.md" ? recipe.skillPath : path.posix.join(path.posix.dirname(recipe.skillPath), file.path) + : file.path; + const entry = byPath.get(assertSkillSnapshotPath(sourcePath)); + if (!entry || entry.type !== "blob" || entry.mode !== (file.executable ? "100755" : "100644") || entry.size !== file.size) { + throw new Error(`The original skill file is unavailable: ${sourcePath}`); + } + return entry; + }); + const files: CompanySkillVersionFileInventoryEntry[] = []; + for (let start = 0; start < entries.length;) { + signal.throwIfAborted(); + let end = start, batchBytes = 0; + while (end < entries.length && end - start < MAX_REPOSITORY_FILES + && batchBytes + manifest.files[end]!.size <= MAX_REPOSITORY_BYTES) { + batchBytes += manifest.files[end]!.size; + end++; + } + if (end === start) throw new Error("Restored skill file exceeds the Git read limit"); + const batch = entries.slice(start, end); + const blobs = snapshot.readBlobs ? await snapshot.readBlobs(batch, signal) : null; + for (let index = start; index < end; index++) { + signal.throwIfAborted(); + const entry = entries[index]!, expected = manifest.files[index]!; + const bytes = blobs?.get(entry.sha) ?? await snapshot.readBlob(entry, signal); + if (bytes.length !== expected.size || packageSha256(bytes) !== expected.sha256) throw new Error(`Restored skill file checksum mismatch: ${expected.path}`); + files.push({ path: expected.path, kind: expected.kind, content: bytes.toString(expected.encoding), + ...(expected.encoding === "base64" ? { encoding: expected.encoding } : {}), ...(expected.executable ? { executable: true } : {}) }); + } + start = end; + } + if (skillPackageManifest(files).contentHash !== manifest.contentHash) throw new Error("Restored skill package checksum mismatch"); + return files; + } finally { await snapshot.release(); } +} diff --git a/server/src/services/skill-package-runtime.ts b/server/src/services/skill-package-runtime.ts new file mode 100644 index 00000000000..cdcf0e25524 --- /dev/null +++ b/server/src/services/skill-package-runtime.ts @@ -0,0 +1,120 @@ +import { measureRunPhase } from "./execution-preparation-timing.js"; +import { promises as fs } from "node:fs"; +import path from "node:path"; +import type { Db } from "@paperclipai/db"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { skillPackageStore, type StoredSkillVersion } from "./skill-package-store.js"; +import { packageSha256, verifySkillPackageDirectory, relocateSkillPackageVerification, type SkillPackageFile, type SkillPackageManifest } from "./skill-package-files.js"; +import { REPOSITORY_CONTENT_DIR, repositorySkillProjection } from "./skill-repository-package.js"; +import { assertSkillSnapshotPath, skillFileBytes } from "./skill-snapshot.js"; +import { ensureSkillCacheDirectory } from "./skill-version-cache.js"; + +type ProjectedFile = SkillPackageFile & { sourcePath?: string; generated?: Buffer }; +// Only the wrapper/policy bytes and file metadata are retained, never a decoded repository. +const projections = new Map(); +let projectionBytes = 0; +const MAX_PROJECTION_BYTES = 16 * 1024 * 1024; +const MAX_PROJECTIONS = 256; +const repositoryFiles = new WeakMap(); +function originalFiles(version: StoredSkillVersion, manifest: SkillPackageManifest): ProjectedFile[] { + if (version.repositorySnapshotId) { + let shared = repositoryFiles.get(manifest); + if (!shared) { + shared = manifest.files.map(file => ({ ...file, sourcePath: file.path, path: `${REPOSITORY_CONTENT_DIR}/${file.path}` })); + repositoryFiles.set(manifest, shared); + } + return [...shared]; + } + if (!version.packageFileMap) return manifest.files.map(file => ({ ...file, sourcePath: file.path })); + const byPath = new Map(manifest.files.map(file => [file.path, file])); + return version.packageFileMap.map(mapping => { + assertSkillSnapshotPath(mapping.path); + assertSkillSnapshotPath(mapping.sourcePath); + const file = byPath.get(mapping.sourcePath); + if (!file) throw new Error("Skill version references a missing package file"); + return { ...file, ...mapping, encoding: mapping.encoding ?? file.encoding }; + }); +} +export function skillPackageRuntime(db: Db) { + const store = skillPackageStore(db); + async function inventory(version: StoredSkillVersion) { + const id = version.repositorySnapshotId ?? version.packageSnapshotId; + if (!id) throw new Error("Skill version has no package"); + const manifest = await store.manifest(version.companyId, id); + const files = originalFiles(version, manifest).map(({ path, kind }) => ({ path, kind })); + if (version.repositorySnapshotId) { + files.unshift({ path: "SKILL.md", kind: "skill" }); + const policy = path.posix.join(path.posix.dirname(version.repositorySkillPath!), "agents/openai.yaml"); + const file = manifest.files.find(file => file.path === policy); + if (file) files.push({ path: "agents/openai.yaml", kind: file.kind }); + } + return files; + } + async function projection(version: StoredSkillVersion): Promise { + const id = version.repositorySnapshotId ?? version.packageSnapshotId; + if (!id) throw new Error("Skill version has no package"); + const manifest = await store.manifest(version.companyId, id); + const key = `${version.companyId}:${version.id}:${manifest.contentHash}`; + const files = originalFiles(version, manifest); + const cached = projections.get(key); + if (cached) return [...files, ...cached.files]; + const generatedFiles: ProjectedFile[] = []; + if (version.repositorySnapshotId) { + if (!version.repositorySkillPath) throw new Error("Repository skill entrypoint is missing"); + const generated = await store.withPackage(version.companyId, id, async (root, manifest) => { + const wanted = new Set([version.repositorySkillPath!, path.posix.join(path.posix.dirname(version.repositorySkillPath!), "agents/openai.yaml")]); + const selected: CompanySkillVersionFileInventoryEntry[] = []; + for (const file of manifest.files.filter(file => wanted.has(file.path))) selected.push({ + ...file, content: (await fs.readFile(path.join(root, file.path))).toString(file.encoding), + }); + return repositorySkillProjection(selected, version.repositorySkillPath!).filter(file => !file.path.startsWith(`${REPOSITORY_CONTENT_DIR}/`)); + }); + for (const file of generated) { + const bytes = skillFileBytes(file); + generatedFiles.push({ path: file.path, kind: file.kind, encoding: file.encoding ?? "utf8", executable: Boolean(file.executable), + size: bytes.length, sha256: packageSha256(bytes), generated: bytes }); + } + } + // Retain only tiny generated wrappers/policies per version. Shared package + // file metadata follows the bounded manifest cache through the WeakMap. + const bytes = generatedFiles.reduce((sum, file) => sum + (file.generated?.length ?? 0) + 512, 0); + while (projections.size && (projections.size >= MAX_PROJECTIONS || projectionBytes + bytes > MAX_PROJECTION_BYTES)) { + const first = projections.keys().next().value!; + projectionBytes -= projections.get(first)!.bytes; + projections.delete(first); + } + if (generatedFiles.length && bytes <= MAX_PROJECTION_BYTES) { projections.set(key, { files: generatedFiles, bytes }); projectionBytes += bytes; } + return [...files, ...generatedFiles]; + } + async function materialize(version: StoredSkillVersion, directory: string) { + const files = await measureRunPhase("skills.repository.read", () => projection(version)); + const parent = path.dirname(directory); + await ensureSkillCacheDirectory(parent, true); + if (await measureRunPhase("skills.version.verify", () => verifySkillPackageDirectory(directory, files), { fileCount: files.length })) return directory; + return store.withPackage(version.companyId, (version.repositorySnapshotId ?? version.packageSnapshotId)!, async root => { + const staging = await fs.mkdtemp(path.join(parent, `.${version.id}-`)); + try { + for (const file of files) { + const target = path.join(staging, file.path); + await fs.mkdir(path.dirname(target), { recursive: true }); + if (file.generated) await fs.writeFile(target, file.generated); + else await fs.copyFile(path.join(root, file.sourcePath!), target); + await fs.chmod(target, file.executable ? 0o555 : 0o444); + } + if (!await verifySkillPackageDirectory(staging, files)) throw new Error("Prepared skill version failed integrity verification"); + await fs.rm(directory, { recursive: true, force: true }); + await fs.rename(staging, directory); + relocateSkillPackageVerification(staging, directory, files); + return directory; + } finally { await fs.rm(staging, { recursive: true, force: true }); } + }); + } + async function readFile(version: StoredSkillVersion, relative: string) { + const file = (await projection(version)).find(file => file.path === relative); + if (!file) return null; + const bytes = file.generated ?? await store.withPackage(version.companyId, + (version.repositorySnapshotId ?? version.packageSnapshotId)!, root => fs.readFile(path.join(root, file.sourcePath!))); + return { path: file.path, kind: file.kind, content: bytes.toString(file.encoding), encoding: file.encoding, executable: file.executable }; + } + return { inventory, materialize, readFile }; +} diff --git a/server/src/services/skill-package-store.ts b/server/src/services/skill-package-store.ts new file mode 100644 index 00000000000..075fab6efff --- /dev/null +++ b/server/src/services/skill-package-store.ts @@ -0,0 +1,674 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; +import { recoverSkillPackage, skillPackageRecoveryReader } from "./skill-package-recovery.js"; +import { touchSkillPackage, trimSkillPackageCache } from "./skill-package-cache.js"; +import { AsyncLocalStorage } from "node:async_hooks"; +import { randomUUID } from "node:crypto"; +import { constants, createReadStream, createWriteStream, promises as fs } from "node:fs"; +import path from "node:path"; +import { pipeline } from "node:stream/promises"; +import { and, eq, getTableColumns, inArray, sql } from "drizzle-orm"; +import { companySkillRepositorySnapshots as snapshots, companySkillVersions, companySkillSources, withDedicatedDbConnection, type Db, type SkillPackageFileMapping, type SkillPackageStorageReceipt, type SkillPackageRestoreRecipe } from "@paperclipai/db"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { getStorageService, type StorageService } from "../storage/index.js"; +import { resolvePaperclipInstanceRoot } from "../home-paths.js"; +import { conflict, notFound } from "../errors.js"; +import { assertSkillSnapshotPath, skillFileBytes } from "./skill-snapshot.js"; +import { ensureSkillCacheDirectory, SkillCacheBusy, withSkillVersionCacheLock } from "./skill-version-cache.js"; +import { boundedHashStream, extractSkillPackageArchive, MAX_SKILL_MANIFEST_BYTES, packageSha256, + skillPackageManifest, skillPackageManifestBytes, validateSkillPackageManifest, verifySkillPackageDirectory, relocateSkillPackageVerification, writeSkillPackageArchive, + type SkillPackageManifest } from "./skill-package-files.js"; + +/** External storage is the default; false is an explicit rolling-upgrade escape hatch. */ +export function skillPackageStorageEnabled() { + return process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED !== "false"; +} + +const descriptorColumns = { id: snapshots.id, companyId: snapshots.companyId, contentHash: snapshots.contentHash, + storage: snapshots.storage, upload: snapshots.upload, createdAt: snapshots.createdAt, + restorable: sql`${snapshots.restore} is not null`, restoreRetryAt: snapshots.restoreRetryAt }; +type Descriptor = Pick & { restorable: boolean }; +export const skillVersionMetadataColumns = { ...getTableColumns(companySkillVersions), + fileInventory: sql`'[]'::jsonb` }; +export type StoredSkillVersion = typeof companySkillVersions.$inferSelect; +export function skillVersionFileMap(files: CompanySkillVersionFileInventoryEntry[], manifest: SkillPackageManifest): SkillPackageFileMapping[] | null { + const stored = new Map(manifest.files.map(file => [file.path, file])); + if (files.every(file => stored.get(file.path)?.kind === file.kind && stored.get(file.path)?.encoding === (file.encoding ?? "utf8"))) return null; + return files.map(file => ({ path: file.path, sourcePath: file.path, kind: file.kind, encoding: file.encoding ?? "utf8" })); +} +type ReadScope = { db: Db; descriptors: Map>; durable: Map>; versions: Map }; +const readScope = new AsyncLocalStorage(); +export function withSkillPackageReadScope(db: Db, action: () => Promise): Promise { + return readScope.getStore()?.db === db ? action() : readScope.run({ db, descriptors: new Map(), durable: new Map(), versions: new Map() }, action); +} +const flights = new WeakMap>>(); +function flight(db: Db, key: string, action: () => Promise): Promise { + let pending = flights.get(db); + if (!pending) { pending = new Map(); flights.set(db, pending); } + const existing = pending.get(key); + if (existing) return existing as Promise; + const work = action().finally(() => pending!.delete(key)); + pending.set(key, work); + return work; +} +type SharedFlight = { work: Promise; controller: AbortController; waiters: number }; +const sharedFlights = new WeakMap>(); +function sharedFlight(db: Db, key: string, cancellation: AbortSignal | undefined, action: (signal: AbortSignal) => Promise): Promise { + cancellation?.throwIfAborted(); + let pending = sharedFlights.get(db); + if (!pending) { pending = new Map(); sharedFlights.set(db, pending); } + let entry = pending.get(key); + if (!entry) { + const controller = new AbortController(); + entry = { controller, waiters: 0, work: Promise.resolve() }; + const created = entry; + entry.work = action(controller.signal).finally(() => { + if (pending!.get(key) === created) pending!.delete(key); + }); + pending.set(key, entry); + } + const current = entry; + current.waiters++; + return new Promise((resolve, reject) => { + let settled = false; + const release = () => { + if (settled) return false; + settled = true; + cancellation?.removeEventListener("abort", aborted); + current.waiters--; + return true; + }; + const aborted = () => { + if (!release()) return; + if (current.waiters) { reject(cancellation!.reason); return; } + if (pending!.get(key) === current) pending!.delete(key); + current.controller.abort(cancellation!.reason); + // The last caller owns cancellation and waits for storage/queue cleanup. + void current.work.then(() => reject(cancellation!.reason), () => reject(cancellation!.reason)); + }; + cancellation?.addEventListener("abort", aborted, { once: true }); + current.work.then(value => { if (release()) resolve(value as T); }, error => { if (release()) reject(error); }); + }); +} +type PackageIoQueue = { active: number; waiting: Array<() => void> }; +const storageIo: PackageIoQueue = { active: 0, waiting: [] }; +// A decoded cold response remains owned until its post-fill cleanup finishes. +const coldResponses: PackageIoQueue = { active: 0, waiting: [] }; +// Bound local response buffers independently so stalled storage does not prevent +// verified cached reads, including reads inside an existing DB transaction. +const inventoryIo: PackageIoQueue = { active: 0, waiting: [] }; +async function withPackageIo(action: () => Promise, signal: AbortSignal, queue = storageIo): Promise { + signal.throwIfAborted(); + if (queue.active >= 2) { + if (queue.waiting.length >= 32) throw conflict("Skill package storage is busy; retry shortly."); + await new Promise((resolve, reject) => { + const ready = () => { signal.removeEventListener("abort", aborted); resolve(); }; + const aborted = () => { + const index = queue.waiting.indexOf(ready); + if (index >= 0) queue.waiting.splice(index, 1); + reject(signal.reason); + }; + queue.waiting.push(ready); + signal.addEventListener("abort", aborted, { once: true }); + }); + } else queue.active++; + try { signal.throwIfAborted(); return await action(); } + finally { const next = queue.waiting.shift(); if (next) next(); else queue.active--; } +} +const manifests = new Map(); +let manifestBytes = 0; +const MAX_MANIFEST_CACHE = 16 * 1024 * 1024; +function rememberManifest(key: string, manifest: SkillPackageManifest, bytes: number) { + if (bytes > MAX_MANIFEST_CACHE) return; + const previous = manifests.get(key); + if (previous) manifestBytes -= previous.bytes; + manifests.delete(key); + while (manifests.size && manifestBytes + bytes > MAX_MANIFEST_CACHE) { + const first = manifests.keys().next().value!; + manifestBytes -= manifests.get(first)!.bytes; + manifests.delete(first); + } + manifests.set(key, { manifest, bytes }); + manifestBytes += bytes; +} +export function skillPackagePaths(companyId: string, snapshotId: string) { + if (![companyId, snapshotId].every(id => /^[a-f0-9-]{36}$/i.test(id))) throw new Error("Invalid skill package identity"); + const parent = path.join(resolvePaperclipInstanceRoot(), "skills", companyId, "__packages__"); + return { parent, directory: path.join(parent, snapshotId), manifest: path.join(parent, `${snapshotId}.json`) }; +} +async function readBoundedObject(storage: StorageService, companyId: string, key: string, length: number, hash: string, signal: AbortSignal) { + if (!Number.isSafeInteger(length) || length < 0 || length > MAX_SKILL_MANIFEST_BYTES) throw new Error("Invalid skill manifest size"); + const object = await storage.getObject(companyId, key, { signal }); + const chunks: Buffer[] = []; + let bytes = 0; + try { + for await (const chunk of object.stream) { + const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); + bytes += buffer.length; + if (bytes > length) throw new Error("Skill manifest exceeds its recorded size"); + chunks.push(buffer); + } + } finally { object.stream.destroy(); } + const result = Buffer.concat(chunks); + if (bytes !== length || packageSha256(result) !== hash) throw new Error("Skill manifest checksum mismatch"); + return result; +} + +/** Owns immutable package storage. Authorization belongs to the referencing + * skill/source, and must be checked even when package bytes are already cached. + * Transaction readers supply their originating DB as cacheOwner for dedicated + * cache locks; metadata queries remain on the caller's transaction. */ +export function skillPackageStore(db: Db, suppliedStorage?: StorageService, cacheOwner: Db = db) { + const storage = () => suppliedStorage ?? getStorageService(); + const scope = (companyId: string, id: string) => and(eq(snapshots.companyId, companyId), eq(snapshots.id, id)); + async function descriptor(companyId: string, id: string): Promise { + const load = async () => { + const [row] = await db.select(descriptorColumns).from(snapshots).where(scope(companyId, id)); + if (!row || (!row.storage && !row.restorable && row.upload && !row.upload.legacy)) throw notFound("Installed skill package is unavailable"); + return row; + }; + const context = readScope.getStore(); + if (context?.db !== db) return load(); + const key = `${companyId}:${id}`; + if (!context.descriptors.has(key)) context.descriptors.set(key, load()); + return context.descriptors.get(key)!; + } + async function version(companyId: string, skillId: string, id: string): Promise { + const context = readScope.getStore(); + const cached = context?.db === db ? context.versions.get(`${companyId}:${id}`) : null; + if (cached) return cached.companySkillId === skillId ? cached : null; + const [row] = await db.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.companySkillId, skillId), eq(companySkillVersions.id, id))); + return row ?? null; + } + async function prefetch(companyId: string, versionIds: string[]) { + const context = readScope.getStore(); + if (context?.db !== db || versionIds.length < 2) return; + const rows = await db.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, companyId), inArray(companySkillVersions.id, [...new Set(versionIds)]))); + for (const row of rows) context.versions.set(`${companyId}:${row.id}`, row); + const ids = [...new Set(rows.flatMap(row => [row.repositorySnapshotId, row.packageSnapshotId]).filter((id): id is string => Boolean(id)))]; + if (!ids.length) return; + const packages = await db.select(descriptorColumns).from(snapshots).where(and(eq(snapshots.companyId, companyId), inArray(snapshots.id, ids))); + for (const row of packages) if (row.storage || row.restorable || !row.upload || row.upload.legacy) { + context.descriptors.set(`${companyId}:${row.id}`, Promise.resolve(row)); + } + } + async function restoreRecipe(row: Descriptor, database = db) { + const [saved] = await database.select({ restore: snapshots.restore }).from(snapshots).where(scope(row.companyId, row.id)); + const recipe = saved?.restore; + if (!recipe || recipe.version !== 1) throw notFound("This skill package has no exact-commit restore recipe"); + const manifest = validateSkillPackageManifest(recipe.manifest); + if (manifest.contentHash !== row.contentHash || (row.storage && packageSha256(skillPackageManifestBytes(manifest)) !== row.storage.manifestSha256)) { + throw new Error("Skill package restore recipe does not match its receipt"); + } + return recipe; + } + async function acquireFiles(row: Descriptor, signal: AbortSignal, database = db) { + signal.throwIfAborted(); + const [current] = await database.select({ retryAt: snapshots.restoreRetryAt }).from(snapshots).where(scope(row.companyId, row.id)); + if (current?.retryAt && current.retryAt.getTime() > Date.now()) throw conflict("The original skill package could not be acquired. Reconnect its repository and retry after the recovery backoff."); + try { + const recipe = await restoreRecipe(row, database); + const read = await skillPackageRecoveryReader(database, row.companyId, recipe); + const files = await recoverSkillPackage(recipe, read, signal); + await database.update(snapshots).set({ restoreRetryAt: null }).where(scope(row.companyId, row.id)); + return files; + } catch (error) { + if (!signal.aborted) await database.update(snapshots).set({ restoreRetryAt: new Date(Date.now() + 5 * 60_000) }).where(scope(row.companyId, row.id)); + throw error; + } + } + async function legacyFiles(row: Descriptor, database = db) { + const [legacy] = await database.select({ files: snapshots.fileInventory, storage: snapshots.storage, upload: snapshots.upload }) + .from(snapshots).where(scope(row.companyId, row.id)); + if (!legacy || (!legacy.files.length && !legacy.storage)) throw notFound("Installed skill package has no saved contents"); + // A reader may have loaded its descriptor just before a migration clears + // the legacy payload. Upgrade this request's descriptor to the published + // receipt instead of reporting a missing historical version. + if (legacy.storage) { row.storage = legacy.storage; row.upload = legacy.upload; } + return legacy.files; + } + async function manifestFor(row: Descriptor, cancellation?: AbortSignal): Promise { + cancellation?.throwIfAborted(); + const paths = skillPackagePaths(row.companyId, row.id); + const key = `${paths.manifest}:${row.storage?.manifestSha256 ?? row.contentHash}`; + const cached = manifests.get(key); + if (cached) return cached.manifest; + const load = async (sharedSignal: AbortSignal) => { + const signal = AbortSignal.any([sharedSignal, AbortSignal.timeout(180_000)]); + await ensureSkillCacheDirectory(paths.parent, true); + // Legacy manifests have no independently stored checksum. Derive them once + // per process from the authoritative row until the operator migrates it. + let bytes: Buffer | null = null; + if (row.storage) { + try { + const file = await fs.open(paths.manifest, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await file.stat(); + if (stat.isFile() && stat.size <= MAX_SKILL_MANIFEST_BYTES) bytes = await file.readFile(); + } finally { await file.close(); } + } catch (error) { + if (!["ENOENT", "ELOOP", "EISDIR"].includes((error as NodeJS.ErrnoException).code ?? "")) throw error; + } + } + if (row.storage) { + if (!bytes || packageSha256(bytes) !== row.storage.manifestSha256) { + try { + if (row.storage.provider !== storage().provider) throw new Error("Skill package storage provider changed"); + bytes = await withPackageIo(() => readBoundedObject(storage(), row.companyId, row.storage!.manifestKey, + row.storage!.manifestBytes, row.storage!.manifestSha256, signal), signal); + } catch (error) { + signal.throwIfAborted(); + const saved = await withPackageIo(() => withDedicatedDbConnection(cacheOwner, async database => { + const [legacy] = await database.select({ files: snapshots.fileInventory }).from(snapshots).where(scope(row.companyId, row.id)); + return legacy?.files.length ? skillPackageManifestBytes(skillPackageManifest(legacy.files)) : null; + }), signal); + if (saved && packageSha256(saved) === row.storage!.manifestSha256) bytes = saved; + else if (row.restorable) bytes = skillPackageManifestBytes((await restoreRecipe(row)).manifest); + else throw error; + } + } + } else if (row.restorable) { + bytes = skillPackageManifestBytes((await restoreRecipe(row)).manifest); + } else if (!bytes) { + bytes = await withPackageIo(() => withDedicatedDbConnection(cacheOwner, async database => { + const files = await legacyFiles(row, database); + return row.storage ? null : skillPackageManifestBytes(skillPackageManifest(files)); + }), signal); + if (!bytes) return manifestFor(row, signal); + } + signal.throwIfAborted(); + if (bytes.length > MAX_SKILL_MANIFEST_BYTES) throw new Error("Skill manifest exceeds its byte bound"); + const manifest = validateSkillPackageManifest(JSON.parse(bytes.toString("utf8"))); + if (row.storage && (manifest.contentHash !== row.contentHash || manifest.files.length !== row.storage.fileCount + || manifest.totalBytes !== row.storage.totalBytes)) throw new Error("Skill package receipt does not match its manifest"); + await ensureSkillCacheDirectory(paths.parent, true); + const temp = `${paths.manifest}.${randomUUID()}`; + try { await fs.writeFile(temp, bytes, { flag: "wx", mode: 0o600, signal }); signal.throwIfAborted(); await fs.rename(temp, paths.manifest); } + finally { await fs.rm(temp, { force: true }); } + rememberManifest(key, manifest, bytes.length); + return manifest; + }; + return sharedFlight(db, `manifest:${key}`, cancellation, load); + } + async function restoreInto(row: Descriptor, manifest: SkillPackageManifest, staging: string, + archivePath: string, signal: AbortSignal, allowRecovery = true, database = db) { + let legacy: CompanySkillVersionFileInventoryEntry[] | null = null; + if (!row.storage && !row.restorable) legacy = await legacyFiles(row, database); + try { + if (row.storage) { + if (row.storage.provider !== storage().provider) throw new Error("Skill package storage provider changed"); + const object = await storage().getObject(row.companyId, row.storage.archiveKey, { signal }); + const digest = boundedHashStream(row.storage.archiveBytes); + await pipeline(object.stream, digest.stream, createWriteStream(archivePath, { flags: "wx", mode: 0o600 }), { signal }); + const actual = digest.result(); + if (actual.bytes !== row.storage.archiveBytes || actual.sha256 !== row.storage.archiveSha256) throw new Error("Skill package archive checksum mismatch"); + await extractSkillPackageArchive(archivePath, staging, manifest, signal); + } + } catch (error) { + signal.throwIfAborted(); + // A missing/corrupt archive must not hide a still-authoritative legacy + // body. Only then use the saved, authorized immutable Git recipe. + const [saved] = await database.select({ files: snapshots.fileInventory }).from(snapshots).where(scope(row.companyId, row.id)); + if (saved?.files.length && skillPackageManifest(saved.files).contentHash === row.contentHash) legacy = saved.files; + else if (row.restorable && allowRecovery) legacy = await acquireFiles(row, signal, database); + else throw error; + await fs.rm(staging, { recursive: true, force: true }); + await fs.mkdir(staging); + } + if (!row.storage && row.restorable && !legacy) { + if (!allowRecovery) throw notFound("Skill package bytes are not stored"); + legacy = await acquireFiles(row, signal, database); + } + if (legacy) { + for (const file of legacy) { + signal.throwIfAborted(); + const target = path.join(staging, assertSkillSnapshotPath(file.path)); + await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); + await fs.writeFile(target, skillFileBytes(file), { flag: "wx", mode: file.executable ? 0o555 : 0o444 }); + } + } + if (!await verifySkillPackageDirectory(staging, manifest.files)) throw new Error("Restored skill package does not match its manifest"); + } + // Archive reads and verification own private scratch space outside the warm + // preparation lock. Administrative reads briefly try that lock to publish. + async function withIsolatedPackage(companyId: string, id: string, + consume: (root: string, manifest: SkillPackageManifest) => Promise, allowRecovery = true, cancellation?: AbortSignal, cacheRead = false) { + const signal = cancellation ? AbortSignal.any([cancellation, AbortSignal.timeout(180_000)]) : AbortSignal.timeout(180_000); + signal.throwIfAborted(); + // Preserve the caller's transaction view before taking an I/O permit. All + // later SQL uses a private connection, so a queued transaction cannot hold + // the ordinary pool connection needed by an active cold response. + const row = await descriptor(companyId, id); + const manifest = await manifestFor(row, signal); + const read = () => withDedicatedDbConnection(cacheOwner, database => + readIsolatedPackage(row, manifest, consume, allowRecovery, signal, cacheRead, database)); + return cacheRead ? withPackageIo(read, signal, coldResponses) : read(); + } + async function readIsolatedPackage(row: Descriptor, manifest: SkillPackageManifest, + consume: (root: string, manifest: SkillPackageManifest) => Promise, allowRecovery: boolean, signal: AbortSignal, cacheRead: boolean, + database: Db) { + const { companyId, id } = row; + let published = false; + const result = await withPackageIo(async () => { + const parent = skillPackagePaths(companyId, id).parent; + await ensureSkillCacheDirectory(parent, true); + const staging = await fs.mkdtemp(path.join(parent, ".verify-")); + const archive = `${staging}.tar.gz`; + try { + await restoreInto(row, manifest, staging, archive, signal, allowRecovery, database); + const result = await consume(staging, manifest); + if (cacheRead) { + // Network, extraction, hashing and response reads all happen before + // this short publication lock. Busy preparation keeps its priority. + const paths = skillPackagePaths(companyId, id); + const previous = await fs.mkdtemp(path.join(parent, ".verify-")); + try { + await withSkillVersionCacheLock(cacheOwner, companyId, async () => { + let moved = false; + try { await fs.rename(paths.directory, previous); moved = true; } + catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } + try { await fs.rename(staging, paths.directory); } + catch (error) { if (moved) await fs.rename(previous, paths.directory); throw error; } + relocateSkillPackageVerification(staging, paths.directory, manifest.files); + touchSkillPackage(paths.directory); + published = true; + }, true); + } catch (error) { if (!(error instanceof SkillCacheBusy)) throw error; } + finally { await fs.rm(previous, { recursive: true, force: true }); } + } + return result; + } finally { await fs.rm(staging, { recursive: true, force: true }); await fs.rm(archive, { force: true }); } + }, signal); + if (published) { + // Verify possible eviction candidates outside both the preparation lock + // and the storage slot used for this fill; take the lock only to remove. + const reader = skillPackageStore(database, storage(), cacheOwner); + await trimSkillPackageCache(database, companyId, skillPackagePaths(companyId, id).directory, + (id, signal) => reader.verifyDurable(companyId, id, signal).catch(() => false), + async remove => { + try { await withSkillVersionCacheLock(cacheOwner, companyId, remove, true); } + catch (error) { if (!(error instanceof SkillCacheBusy)) throw error; } + }, signal); + } + signal.throwIfAborted(); + return result; + } + async function materialize(row: Descriptor, manifest: SkillPackageManifest) { + const paths = skillPackagePaths(row.companyId, row.id); + return flight(db, `files:${paths.directory}`, async () => { + await ensureSkillCacheDirectory(paths.parent, true); + if (await verifySkillPackageDirectory(paths.directory, manifest.files)) return { directory: paths.directory, created: false }; + const signal = AbortSignal.timeout(180_000); + return withPackageIo(async () => { + const staging = await fs.mkdtemp(path.join(paths.parent, ".package-")); + const archivePath = `${staging}.tar.gz`; + try { + await restoreInto(row, manifest, staging, archivePath, signal); + // All consumers hold the company cache lock, including processes sharing a volume. + await fs.rm(paths.directory, { recursive: true, force: true }); + await fs.rename(staging, paths.directory); + relocateSkillPackageVerification(staging, paths.directory, manifest.files); + return { directory: paths.directory, created: true }; + } finally { await fs.rm(staging, { recursive: true, force: true }); await fs.rm(archivePath, { force: true }); } + }, signal); + }); + } + async function withPackage(companyId: string, id: string, consume: (root: string, manifest: SkillPackageManifest) => Promise) { + const row = await descriptor(companyId, id); + const manifest = await manifestFor(row); + return withSkillVersionCacheLock(db, companyId, async () => { + const restored = await materialize(row, manifest); + touchSkillPackage(restored.directory); + if (restored.created) await trimSkillPackageCache(db, companyId, restored.directory, + (id, signal) => verifyDurable(companyId, id, signal).catch(() => false)); + return consume(restored.directory, manifest); + }); + } + async function files(companyId: string, id: string, isolated = false): Promise { + return (isolated ? withIsolatedPackage : withPackage)(companyId, id, async (root, manifest) => { + const result: CompanySkillVersionFileInventoryEntry[] = []; + for (const entry of manifest.files) result.push({ path: entry.path, kind: entry.kind, content: (await fs.readFile(path.join(root, entry.path))).toString(entry.encoding), + ...(entry.encoding === "base64" ? { encoding: entry.encoding } : {}), ...(entry.executable ? { executable: true } : {}) }); + return result; + }); + } + async function prepare(companyId: string, files: CompanySkillVersionFileInventoryEntry[], migrateLegacy = false, cancellation?: AbortSignal, restore?: SkillPackageRestoreRecipe): Promise { + cancellation?.throwIfAborted(); + const manifest = skillPackageManifest(files); + if (restore && !skillPackageManifestBytes(restore.manifest).equals(skillPackageManifestBytes(manifest))) throw new Error("Restore recipe does not describe these package bytes"); + const preparedId = await flight(db, `prepare:${companyId}:${manifest.contentHash}`, async () => { + const signal = cancellation ? AbortSignal.any([cancellation, AbortSignal.timeout(180_000)]) : AbortSignal.timeout(180_000); + const token = randomUUID(); + const prefix = `${companyId}/skill-packages/${token}`; + const upload = { token, legacy: false, provider: storage().provider, archiveKey: `${prefix}/files.tar.gz`, manifestKey: `${prefix}/manifest.json`, expiresAt: new Date(Date.now() + 10 * 60_000).toISOString() }; + const [created] = await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); + return tx.insert(snapshots).values({ companyId, contentHash: manifest.contentHash, upload, restore, + migrationAt: null, maintenanceAt: new Date(Date.now() + 24 * 60 * 60_000) }).onConflictDoNothing().returning(descriptorColumns); + }); + let existing = created ?? (await db.select(descriptorColumns).from(snapshots).where(and(eq(snapshots.companyId, companyId), eq(snapshots.contentHash, manifest.contentHash))))[0]!; + if (!created) { + if (existing.storage || (!existing.upload && !migrateLegacy && !existing.restorable)) { + // Content deduplication must not attach a recipe with different kind or + // encoding metadata to an already published manifest. + if (restore && !existing.restorable && existing.storage?.manifestSha256 === packageSha256(skillPackageManifestBytes(manifest))) { + await db.update(snapshots).set({ restore }).where(and(scope(companyId, existing.id), sql`${snapshots.restore} is null`)); + } + return existing.id; + } + // A stale claim is replaced atomically; the DB trigger retains its exact + // keys in the deletion outbox. Never reuse keys from another attempt. + upload.legacy = !existing.upload || Boolean(existing.upload.legacy); + const [claimed] = await db.update(snapshots).set({ upload }).where(and(scope(companyId, existing.id), + sql`${snapshots.storage} is null`, + sql`(${snapshots.upload} is null or (${snapshots.upload}->>'expiresAt')::timestamptz <= now())`)).returning(descriptorColumns); + if (!claimed) throw conflict("This skill package is being saved. Retry after its upload settles."); + existing = claimed; + } + return withPackageIo(async () => { + const root = skillPackagePaths(companyId, existing.id).parent; + await ensureSkillCacheDirectory(root, true); + const staging = await fs.mkdtemp(path.join(root, ".upload-")); + const archivePath = path.join(staging, "files.tar.gz"); + const manifestBuffer = skillPackageManifestBytes(manifest); + if (manifestBuffer.length > MAX_SKILL_MANIFEST_BYTES) throw new Error("Skill manifest exceeds its byte bound"); + const backend = storage(); + try { + const archive = await writeSkillPackageArchive(archivePath, files, signal); + await backend.putFile({ companyId, namespace: "skill-packages", objectKey: upload.archiveKey, originalFilename: "files.tar.gz", + contentType: "application/gzip", body: createReadStream(archivePath), byteSize: archive.bytes, sha256: archive.sha256, signal }); + await backend.putFile({ companyId, namespace: "skill-packages", objectKey: upload.manifestKey, originalFilename: "manifest.json", + contentType: "application/json", body: manifestBuffer, signal }); + const receipt: SkillPackageStorageReceipt = { format: "tar-gzip-v1", provider: backend.provider, ...(!restore ? { durable: true } : {}), + archiveKey: upload.archiveKey, archiveSha256: archive.sha256, archiveBytes: archive.bytes, + manifestKey: upload.manifestKey, manifestSha256: packageSha256(manifestBuffer), manifestBytes: manifestBuffer.length, + fileCount: manifest.files.length, totalBytes: manifest.totalBytes }; + // A complete read verifies storage independently before publishing any reference. + await readBoundedObject(backend, companyId, receipt.manifestKey, receipt.manifestBytes, receipt.manifestSha256, signal); + const verify = await backend.getObject(companyId, receipt.archiveKey, { signal }); + const digest = boundedHashStream(receipt.archiveBytes); + const downloaded = path.join(staging, "verified.tar.gz"); + await pipeline(verify.stream, digest.stream, createWriteStream(downloaded, { flags: "wx", mode: 0o600 }), { signal }); + const actual = digest.result(); + if (actual.bytes !== receipt.archiveBytes || actual.sha256 !== receipt.archiveSha256) throw new Error("Uploaded skill package checksum mismatch"); + const restored = path.join(staging, "restored"); + await fs.mkdir(restored); + await extractSkillPackageArchive(downloaded, restored, manifest, signal); + signal.throwIfAborted(); + const [saved] = await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); + return tx.update(snapshots).set({ storage: receipt, upload: null, ...(restore ? { restore } : {}) }) + .where(and(scope(companyId, existing.id), sql`${snapshots.upload}->>'token' = ${token}`)).returning({ id: snapshots.id }); + }); + if (!saved) throw conflict("Skill package upload ownership changed"); + return saved.id; + } finally { + // Failed/ambiguous uploads retain their durable claim. Maintenance reconciles + // exact owned keys; never delete a successful object on an uncertain SQL result. + await fs.rm(staging, { recursive: true, force: true }); + } + }, signal); + }); + { + const [retained] = await db.update(snapshots).set({ maintenanceAt: new Date(Date.now() + 24 * 60 * 60_000), + ...(!restore ? { storage: sql`jsonb_set(${snapshots.storage}, '{durable}', 'true'::jsonb)` } : {}) }) + .where(and(scope(companyId, preparedId), sql`${snapshots.storage} is not null`)).returning({ id: snapshots.id }); + if (!retained) return prepare(companyId, files, true, cancellation, restore); + } + return preparedId; + } + async function bindRestore(companyId: string, id: string, restore: SkillPackageRestoreRecipe) { + const manifest = validateSkillPackageManifest(restore.manifest); + // Called in the source publication transaction. A failed import must not + // leave a later successful import bound to an uncommitted source identity. + await db.update(snapshots).set({ restore }).where(and(scope(companyId, id), + eq(snapshots.contentHash, manifest.contentHash), + sql`${snapshots.storage}->>'manifestSha256' = ${packageSha256(skillPackageManifestBytes(manifest))}`, + sql`exists (select 1 from ${companySkillSources} s where s.company_id = ${companyId} + and s.id = ${restore.sourceId} and s.removed_at is null)`, + sql`not exists (select 1 from ${companySkillSources} s where s.company_id = ${companyId} + and s.id::text = ${snapshots.restore}->>'sourceId' and s.removed_at is null)`)); + } + async function verifyDurable(companyId: string, id: string, cancellation?: AbortSignal, + validate?: (root: string, manifest: SkillPackageManifest) => Promise): Promise { + const action = async () => { + cancellation?.throwIfAborted(); + const row = await descriptor(companyId, id); + if (!row.storage) { + await manifestFor(row, cancellation); + if (!row.storage) return validate ? withIsolatedPackage(companyId, id, validate, false, cancellation) : true; + } + if (row.storage.provider !== storage().provider) throw new Error("Skill package storage provider changed; migrate its objects before switching storage"); + const signal = cancellation ? AbortSignal.any([cancellation, AbortSignal.timeout(180_000)]) : AbortSignal.timeout(180_000); + return withPackageIo(async () => { + const receipt = row.storage!; + const bytes = await readBoundedObject(storage(), companyId, receipt.manifestKey, receipt.manifestBytes, receipt.manifestSha256, signal); + const manifest = validateSkillPackageManifest(JSON.parse(bytes.toString("utf8"))); + if (manifest.contentHash !== row.contentHash) return false; + const parent = skillPackagePaths(companyId, id).parent; + await ensureSkillCacheDirectory(parent, true); + const staging = await fs.mkdtemp(path.join(parent, ".verify-")); + try { + const object = await storage().getObject(companyId, receipt.archiveKey, { signal }); + const digest = boundedHashStream(receipt.archiveBytes); + const archive = path.join(staging, "files.tar.gz"); + await pipeline(object.stream, digest.stream, createWriteStream(archive, { flags: "wx", mode: 0o600 }), { signal }); + const actual = digest.result(); + if (actual.bytes !== receipt.archiveBytes || actual.sha256 !== receipt.archiveSha256) return false; + const restored = path.join(staging, "files"); + await fs.mkdir(restored); + await extractSkillPackageArchive(archive, restored, manifest, signal); + return validate ? await validate(restored, manifest) : true; + } finally { await fs.rm(staging, { recursive: true, force: true }); } + }, signal); + }; + const context = readScope.getStore(); + const key = `${companyId}:${id}`; + if (context?.db !== db || validate || cancellation) return action(); + if (!context.durable.has(key)) context.durable.set(key, action()); + return context.durable.get(key)!; + } + async function readPackageFile(root: string, entry: SkillPackageManifest["files"][number], signal?: AbortSignal) { + signal?.throwIfAborted(); + const filename = path.join(root, entry.path); + await ensureSkillCacheDirectory(path.dirname(filename)); + const handle = await fs.open(filename, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await handle.stat(); + if (!stat.isFile() || stat.size !== entry.size) throw new Error("Invalid cached skill file"); + const bytes = Buffer.alloc(entry.size); + let offset = 0; + while (offset < bytes.length) { + signal?.throwIfAborted(); + const { bytesRead } = await handle.read(bytes, offset, bytes.length - offset, offset); + if (!bytesRead) throw new Error("Invalid cached skill file"); + offset += bytesRead; + } + if (packageSha256(bytes) !== entry.sha256) throw new Error("Invalid cached skill file"); + return bytes; + } finally { await handle.close(); } + } + async function versionFiles(version: StoredSkillVersion, allowRecovery = true, cancellation?: AbortSignal): Promise { + const signal = cancellation ? AbortSignal.any([cancellation, AbortSignal.timeout(180_000)]) : AbortSignal.timeout(180_000); + signal.throwIfAborted(); + if (!version.packageSnapshotId) { + const [row] = await db.select({ ...skillVersionMetadataColumns, files: companySkillVersions.fileInventory }).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, version.companyId), eq(companySkillVersions.id, version.id))); + if (row?.packageSnapshotId) return versionFiles(row, allowRecovery, signal); + return row?.files ?? []; + } + const id = version.packageSnapshotId; + const manifest = await manifestFor(await descriptor(version.companyId, id), signal); + const read = async (root: string) => { + const byPath = new Map(manifest.files.map(file => [file.path, file])); + const result: CompanySkillVersionFileInventoryEntry[] = []; + for (const mapping of version.packageFileMap ?? manifest.files.map(file => ({ ...file, sourcePath: file.path }))) { + assertSkillSnapshotPath(mapping.path); + assertSkillSnapshotPath(mapping.sourcePath); + const file = byPath.get(mapping.sourcePath); + if (!file) throw new Error("Skill version references a missing package file"); + const encoding = mapping.encoding ?? file.encoding; + result.push({ path: mapping.path, kind: mapping.kind, + content: (await readPackageFile(root, file, signal)).toString(encoding), + ...(encoding === "base64" ? { encoding } : {}), ...(file.executable ? { executable: true } : {}) }); + } + return result; + }; + // Verify bytes as they are read, without holding the warm preparation lock. + // A descriptor survives eviction; a missing/racing file falls back to storage. + const root = skillPackagePaths(version.companyId, id).directory; + const result = await withPackageIo(() => read(root).catch(() => null), signal, inventoryIo); + if (result) { + touchSkillPackage(root); + return result; + } + signal.throwIfAborted(); + const acquire = async () => { + const cached = await withPackageIo(() => read(root).catch(() => null), signal, inventoryIo); + return cached ?? withIsolatedPackage(version.companyId, id, read, allowRecovery, signal, true); + }; + return cancellation ? acquire() : flight(db, `version-read:${version.companyId}:${version.id}:${allowRecovery}`, acquire); + } + async function readVersionFile(version: StoredSkillVersion, relativePath: string, allowRecovery = true) { + assertSkillSnapshotPath(relativePath); + if (!version.packageSnapshotId) return (await versionFiles(version, allowRecovery)).find(file => file.path === relativePath); + const id = version.packageSnapshotId; + const manifest = await manifestFor(await descriptor(version.companyId, id)); + const mapping = version.packageFileMap?.find(file => file.path === relativePath); + if (version.packageFileMap && !mapping) return undefined; + const entry = manifest.files.find(file => file.path === (mapping?.sourcePath ?? relativePath)); + if (!entry) return undefined; + const signal = AbortSignal.timeout(180_000); + const read = async (root: string) => { + const bytes = await readPackageFile(root, entry, signal); + const encoding = mapping?.encoding ?? entry.encoding; + return { path: relativePath, kind: mapping?.kind ?? entry.kind, content: bytes.toString(encoding), + encoding, executable: entry.executable }; + }; + // Immutable cache files can be read without the preparation lock. An open + // descriptor survives eviction; missing/racing files fall back to storage. + const cached = await withPackageIo(() => read(skillPackagePaths(version.companyId, id).directory).catch(() => null), signal, inventoryIo); + if (cached) return cached; + signal.throwIfAborted(); + return flight(db, `file-read:${version.companyId}:${version.id}:${relativePath}:${allowRecovery}`, + async () => { + const cached = await withPackageIo(() => read(skillPackagePaths(version.companyId, id).directory).catch(() => null), signal, inventoryIo); + return cached ?? withIsolatedPackage(version.companyId, id, read, allowRecovery, signal, true); + }); + } + async function versionInventory(version: StoredSkillVersion): Promise> { + if (version.packageSnapshotId) { + const index = version.packageFileMap ?? (await manifestFor(await descriptor(version.companyId, version.packageSnapshotId))).files; + return index.map(({ path, kind }) => ({ path, kind })); + } + const [row] = await db.select({ ...skillVersionMetadataColumns, files: sql>` + coalesce((select jsonb_agg(jsonb_build_object('path', f->>'path', 'kind', f->>'kind')) + from jsonb_array_elements(${companySkillVersions.fileInventory}) f), '[]'::jsonb)` }) + .from(companySkillVersions).where(and(eq(companySkillVersions.companyId, version.companyId), eq(companySkillVersions.id, version.id))); + if (row?.packageSnapshotId) return versionInventory(row); + return row?.files ?? []; + } + return { descriptor, prepare, bindRestore, files, version, prefetch, versionFiles, versionInventory, readVersionFile, withPackage, withIsolatedPackage, verifyDurable, + manifest: async (companyId: string, id: string, cancellation?: AbortSignal) => manifestFor(await descriptor(companyId, id), cancellation) }; +} diff --git a/server/src/services/skill-package-work.test.ts b/server/src/services/skill-package-work.test.ts new file mode 100644 index 00000000000..243c89f022a --- /dev/null +++ b/server/src/services/skill-package-work.test.ts @@ -0,0 +1,176 @@ +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { eq, sql } from "drizzle-orm"; +import { companies, companySkills, companySkillVersions as versions, companySkillRepositorySnapshots as snapshots, skillPackageDeletions, createDb } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { createStorageService } from "../storage/service.js"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { skillPackageWork } from "./skill-package-work.js"; +import { skillPackageStore } from "./skill-package-store.js"; +import { skillSnapshotHash } from "./skill-snapshot.js"; +import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "./task-admission.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; +import { logger } from "../middleware/logger.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("automatic skill package work", () => { + let database: Awaited>; + let db: ReturnType; + let storage: ReturnType; + let coordinator: ReturnType | undefined; + let home: string; + const previousHome = process.env.PAPERCLIP_HOME; + const files = [{ path: "SKILL.md", kind: "skill" as const, content: "---\nname: legacy\n---\noriginal" }]; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("skill-package-work-"); + db = createDb(database.connectionString, { maxConnections: 2 }); + home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-package-work-")); + process.env.PAPERCLIP_HOME = home; + storage = createStorageService(createLocalDiskStorageProvider(path.join(home, "objects"))); + }); + beforeEach(async () => { + await db.execute(sql`truncate companies cascade`); + await db.execute(sql`truncate skill_package_deletions`); + }); + afterEach(async () => { await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks(); }); + afterAll(async () => { + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; + await db?.$client.end({ timeout: 1 }); await database?.cleanup(); + if (home) await fs.rm(home, { recursive: true, force: true }); + }); + async function company() { + const id = randomUUID(); + await db.insert(companies).values({ id, name: "Migration", issuePrefix: "MIG" }); + return id; + } + it("stops querying after empty startup and retains a first committed wake through drain", async () => { + const companyId = await company(); + let draining = false; + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, onError: error => { throw error; } }); + const work = skillPackageWork(db, storage).start(coordinator); + await work.ready; + expect(coordinator.nextWakeAt()).toBeNull(); + let queries = 0; + const debug = db.$client.options.debug; + db.$client.options.debug = () => { queries++; }; + try { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + await vi.advanceTimersByTimeAsync(120_000); + expect(queries).toBe(0); + draining = true; + let id: string; + await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); + [id] = (await tx.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning({ id: snapshots.id })).map(row => row.id); + }); + await vi.advanceTimersByTimeAsync(0); + await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0)); + const baseline = queries; + await vi.advanceTimersByTimeAsync(60_000); + expect(queries).toBe(baseline); + draining = false; + work.wake(); + await vi.advanceTimersByTimeAsync(0); + await vi.waitFor(async () => { + const [row] = await db.select().from(snapshots).where(eq(snapshots.id, id!)); + expect(row!.fileInventory).toEqual([]); + expect(row!.storage).not.toBeNull(); + }); + } finally { db.$client.options.debug = debug; } + }); + it("keeps legacy contents on failure, persists a retry and resumes after restart", async () => { + const companyId = await company(); + const [row] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const upload = vi.spyOn(storage, "putFile").mockRejectedValue(new Error("offline")); + const service = skillPackageWork(db, storage); + await service.tick(new AbortController().signal); + const [failed] = await db.select().from(snapshots).where(eq(snapshots.id, row!.id)); + expect(failed!.fileInventory).toEqual(files); + expect(failed!.migrationAt!.getTime()).toBeGreaterThan(Date.now()); + expect(await service.deadline()).toBe(failed!.migrationAt!.getTime()); + upload.mockRestore(); + await db.update(snapshots).set({ migrationAt: new Date(0), upload: { ...failed!.upload!, expiresAt: new Date(0).toISOString() } }).where(eq(snapshots.id, row!.id)); + await skillPackageWork(db, storage).tick(new AbortController().signal); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.id, row!.id)); + expect(saved!.fileInventory).toEqual([]); + expect(saved!.migrationAt).toBeNull(); + expect(await skillPackageStore(db, storage).files(companyId, row!.id)).toEqual(files); + }); + it("reports a safe conversion failure reason without logging the legacy payload", async () => { + const companyId = await company(); + const legacy = [{ ...files[0]!, content: "private payload" }]; + const [row] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: legacy, migrationAt: new Date(0) }).returning(); + const warning = vi.spyOn(logger, "warn").mockImplementation(() => {}); + await skillPackageWork(db, storage).tick(new AbortController().signal); + expect(warning).toHaveBeenCalledWith({ packageId: row!.id, failure: { reason: "legacy_hash_mismatch", errorCode: "unknown" } }, expect.any(String)); + expect(JSON.stringify(warning.mock.calls)).not.toContain("private payload"); + }); + it.each(["manifest.json", "files.tar.gz"])("stops conversion during a stalled %s read and retains legacy bytes", async object => { + const companyId = await company(); + const packageId = await skillPackageStore(db, storage).prepare(companyId, files); + const [skill] = await db.insert(companySkills).values({ companyId, key: "legacy", slug: "legacy", name: "Legacy", markdown: files[0]!.content }).returning(); + const [version] = await db.insert(versions).values({ companyId, companySkillId: skill!.id, revisionNumber: 1, fileInventory: files, migrationAt: new Date(0), + ...(object === "manifest.json" ? { repositorySnapshotId: packageId, repositorySkillPath: "SKILL.md" } : { packageSnapshotId: packageId }) }).returning(); + const get = storage.getObject.bind(storage); + let signalStarted!: () => void; + const started = new Promise(resolve => { signalStarted = resolve; }); + let pendingSignal: AbortSignal | undefined; + let releasePending: () => void = () => {}; + vi.spyOn(storage, "getObject").mockImplementation(async (companyId, key, options) => { + if (!key.endsWith(object)) return get(companyId, key, options); + pendingSignal = options?.signal; + signalStarted(); + return new Promise((_resolve, reject) => { + releasePending = () => reject(new Error("Test released stalled read")); + pendingSignal?.addEventListener("abort", () => reject(pendingSignal!.reason), { once: true }); + }); + }); + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: () => {} }); + skillPackageWork(db, storage).start(coordinator); + await started; + const stopping = coordinator.stop(); + let timeout: ReturnType | undefined; + try { + await Promise.race([stopping, new Promise((_resolve, reject) => { + timeout = setTimeout(() => reject(new Error("Conversion did not stop promptly")), 1_000); + })]); + expect(pendingSignal?.aborted).toBe(true); + const [saved] = await db.select().from(versions).where(eq(versions.id, version!.id)); + expect(saved!.fileInventory).toEqual(files); + expect(saved!.migrationAt).not.toBeNull(); + } finally { clearTimeout(timeout); releasePending(); await stopping; } + }); + it.each([ + ["deleteObject", "stop"], ["headObject", "stop"], ["deleteObject", "deadline"], ["headObject", "deadline"], + ] as const)("releases a stalled %s on %s and retains its deletion receipt", async (method, cancellation) => { + const companyId = await company(); + const [receipt] = await db.insert(skillPackageDeletions).values({ companyId, provider: "local_disk", + archiveKey: `${companyId}/expired/archive`, manifestKey: `${companyId}/expired/manifest`, deleteAfter: new Date(0) }).returning(); + const provider = createLocalDiskStorageProvider(path.join(home, "objects")); + let signalStarted: () => void = () => {}; + const started = new Promise(resolve => { signalStarted = resolve; }); + let pendingSignal: AbortSignal | undefined; + vi.spyOn(provider, method).mockImplementation(async input => { + pendingSignal = input.signal; + signalStarted(); + return new Promise(() => {}); + }); + const timeout = new AbortController(); + const deadline = vi.spyOn(AbortSignal, "timeout").mockReturnValue(timeout.signal); + vi.spyOn(logger, "warn").mockImplementation(() => {}); + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: () => {} }); + const work = skillPackageWork(db, createStorageService(provider)).start(coordinator); + await started; + expect(deadline).toHaveBeenCalledWith(30_000); + if (cancellation === "stop") await coordinator.stop(); + else { timeout.abort(new Error("Storage deadline exceeded")); await work.ready; } + expect(pendingSignal?.aborted).toBe(true); + const [saved] = await db.select().from(skillPackageDeletions).where(eq(skillPackageDeletions.id, receipt!.id)); + expect(saved).toBeDefined(); + if (cancellation === "deadline") expect(saved!.deleteAfter.getTime()).toBeGreaterThan(Date.now()); + }); +}); diff --git a/server/src/services/skill-package-work.ts b/server/src/services/skill-package-work.ts new file mode 100644 index 00000000000..ba6bbf80ec8 --- /dev/null +++ b/server/src/services/skill-package-work.ts @@ -0,0 +1,191 @@ +import { readPaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils"; +import { promises as fs } from "node:fs"; +import { and, eq, isNotNull, isNull, lte, sql } from "drizzle-orm"; +import { companySkillRepositorySnapshots as snapshots, companySkillVersions as versions, + skillPackageDeletions, agents, type Db } from "@paperclipai/db"; +import { getStorageService, type StorageService } from "../storage/index.js"; +import { storageFailureDiagnosticsFor } from "../storage/failure-diagnostics.js"; +import { logger } from "../middleware/logger.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { validateSkillPackageManifest, skillPackageManifestBytes, packageSha256 } from "./skill-package-files.js"; +import { skillPackageMaintenance } from "./skill-package-maintenance.js"; +import { skillPackagePaths, skillPackageStorageEnabled } from "./skill-package-store.js"; +import { ensureSkillCacheDirectory, protectedVersions, SkillCacheBusy, withSkillVersionCacheLock } from "./skill-version-cache.js"; + +export const SKILL_PACKAGE_RETENTION_MS = 30 * 24 * 60 * 60_000; +const RETRY_MS = 60 * 60_000; +const PAGE_SIZE = 4; + +function failureDetails(error: unknown) { + const storage = storageFailureDiagnosticsFor(error); + if (storage) return { reason: "storage_failure", ...storage }; + const own = (key: string) => error && typeof error === "object" ? Object.getOwnPropertyDescriptor(error, key)?.value : undefined; + const message = own("message"); + const code = own("code") ?? own("name"); + const reasons = [ + ["Legacy package hash mismatch", "legacy_hash_mismatch"], + ["Legacy version differs", "legacy_version_mismatch"], + ["Package verification failed", "verification_failed"], + ["Skill package file checksum mismatch", "checksum_mismatch"], + ["Skill package archive checksum mismatch", "checksum_mismatch"], + ["Invalid skill restore recipe", "invalid_restore_recipe"], + ["Object not found", "object_missing"], + ["Storage did not confirm package deletion", "deletion_unconfirmed"], + ] as const; + return { reason: reasons.find(([prefix]) => typeof message === "string" && message.startsWith(prefix))?.[1] ?? "operation_failed", + errorCode: typeof code === "string" && /^[A-Za-z0-9._-]{1,64}$/.test(code) ? code : "unknown" }; +} + +/** A finite upgrade plus persisted cleanup deadlines. No work is added to turn + * preparation; completed history has no recurring SQL/network obligation. */ +export function skillPackageWork(db: Db, storage?: StorageService) { + const maintenance = skillPackageMaintenance(db, storage); + const backend = () => storage ?? getStorageService(); + async function retire(id: string, signal: AbortSignal) { + const [row] = await db.select({ id: snapshots.id, companyId: snapshots.companyId, contentHash: snapshots.contentHash, createdAt: snapshots.createdAt, + storage: snapshots.storage, upload: snapshots.upload, restore: snapshots.restore, migrationAt: snapshots.migrationAt }) + .from(snapshots).where(eq(snapshots.id, id)); + if (!row) return; + signal.throwIfAborted(); + // Orphan deletion is delayed so an upload can publish its first version. + const removeLocal = async () => { + const paths = skillPackagePaths(row.companyId, id); + try { + await ensureSkillCacheDirectory(paths.parent); + await fs.rm(paths.directory, { recursive: true, force: true }); + await fs.rm(paths.manifest, { force: true }); + } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } + }; + if (await withSkillVersionCacheLock(db, row.companyId, async () => { + if (!(await maintenance.collectOrphans({ id, apply: true, signal })).removed) return false; + await removeLocal(); + return true; + }, true)) return; + const orphanAt = row.createdAt.getTime() + 24 * 60 * 60_000; + let next: Date | null = orphanAt > Date.now() ? new Date(orphanAt) : null; + if (row.upload) next = new Date(Math.max(Date.now() + RETRY_MS, new Date(row.upload.expiresAt).getTime(), row.createdAt.getTime() + 24 * 60 * 60_000)); + else if (row.restore && row.storage && !row.storage.durable) { + const expires = row.createdAt.getTime() + SKILL_PACKAGE_RETENTION_MS; + if (expires > Date.now()) next = new Date(expires); + else if (!row.migrationAt) { + const manifest = validateSkillPackageManifest(row.restore.manifest); + if (row.restore.version !== 1 || manifest.contentHash !== row.contentHash + || packageSha256(skillPackageManifestBytes(manifest)) !== row.storage.manifestSha256) throw new Error("Invalid skill restore recipe; keep durable bytes"); + const publishing = await db.transaction(async tx => { + // Publication takes this row lock before making a version current. + // Skip active publishers before taking the warm preparation lock. + const [latest] = await tx.select({ storage: snapshots.storage, upload: snapshots.upload, + migrationAt: snapshots.migrationAt, maintenanceAt: snapshots.maintenanceAt }) + .from(snapshots).where(eq(snapshots.id, id)).for("update", { skipLocked: true }); + if (!latest) return true; + if (!latest.storage || latest.storage.durable || latest.upload || latest.migrationAt + || !latest.maintenanceAt || latest.maintenanceAt.getTime() > Date.now()) return; + await withSkillVersionCacheLock(db, row.companyId, async () => { + const referenced = await tx.select({ id: versions.id, repositorySnapshotId: versions.repositorySnapshotId }).from(versions).where(and(eq(versions.companyId, row.companyId), + sql`(${versions.repositorySnapshotId} = ${id} or ${versions.packageSnapshotId} = ${id})`)); + if (referenced.some(version => version.repositorySnapshotId !== id)) return; + const [current] = await tx.execute(sql`select 1 from company_skills s where s.company_id = ${row.companyId} + and s.current_version_id in (select id from company_skill_versions where repository_snapshot_id = ${id} or package_snapshot_id = ${id}) + union all select 1 from company_skill_test_runs t where t.company_id = ${row.companyId} + and t.status in ('queued', 'running') and t.skill_version_id in + (select id from company_skill_versions where repository_snapshot_id = ${id} or package_snapshot_id = ${id}) limit 1`); + if (current) return; + const pinned = new Set(); + for (const agent of await tx.select({ config: agents.adapterConfig }).from(agents).where(eq(agents.companyId, row.companyId))) { + for (const entry of readPaperclipSkillSyncPreference(agent.config ?? {}).desiredSkillEntries) if (entry.versionId) pinned.add(entry.versionId); + } + const { versions: active } = await protectedVersions(tx as unknown as Db, row.companyId); + if (!active || referenced.some(version => active.has(version.id) || pinned.has(version.id))) return; + signal.throwIfAborted(); + // Metadata/recipe remain. Triggered deletion receipts retain the old + // object keys through the rollback grace; future reads reacquire the SHA. + const retired = await tx.update(snapshots).set({ storage: null }).where(and(eq(snapshots.id, id), isNull(snapshots.upload), + sql`${snapshots.storage}->>'durable' is distinct from 'true'`)).returning({ id: snapshots.id }); + if (retired.length) await removeLocal(); + }, true); + return false; + }); + if (publishing) return; + } + } + await db.update(snapshots).set({ maintenanceAt: next }).where(and(eq(snapshots.id, id), lte(snapshots.maintenanceAt, new Date()))); + } + async function tick(signal: AbortSignal) { + if (!skillPackageStorageEnabled()) return; + for (const table of [snapshots, versions]) { + const rows = await db.select({ id: table.id }).from(table).where(lte(table.migrationAt, new Date())).orderBy(table.migrationAt).limit(PAGE_SIZE); + for (const row of rows) { + signal.throwIfAborted(); + try { + if (table === snapshots) await maintenance.migrateSnapshots({ id: row.id, apply: true, clearLegacy: true, signal }); + else { + // Bundled releases intentionally retain their small inline contract. + const result = await maintenance.migrateVersions({ id: row.id, apply: true, clearLegacy: true, signal }); + if (!result.inspected) await db.update(versions).set({ migrationAt: null }).where(eq(versions.id, row.id)); + } + } catch (error) { + signal.throwIfAborted(); + await db.update(table).set({ migrationAt: new Date(Date.now() + RETRY_MS) }).where(and(eq(table.id, row.id), isNotNull(table.migrationAt))); + logger.warn({ packageId: row.id, failure: failureDetails(error) }, "Skill package conversion deferred; original contents retained"); + } + } + } + const due = await db.select({ id: snapshots.id }).from(snapshots).where(lte(snapshots.maintenanceAt, new Date())).orderBy(snapshots.maintenanceAt).limit(PAGE_SIZE); + for (const { id } of due) { + try { await retire(id, signal); } + catch (error) { + signal.throwIfAborted(); + await db.update(snapshots).set({ maintenanceAt: new Date(Date.now() + (error instanceof SkillCacheBusy ? 60_000 : RETRY_MS)) }).where(and(eq(snapshots.id, id), lte(snapshots.maintenanceAt, new Date()))); + if (!(error instanceof SkillCacheBusy)) logger.warn({ packageId: id, failure: failureDetails(error) }, "Skill package cleanup deferred"); + } + } + const deletions = await db.select({ id: skillPackageDeletions.id }).from(skillPackageDeletions).where(and( + eq(skillPackageDeletions.provider, backend().provider), lte(skillPackageDeletions.deleteAfter, new Date()))).limit(PAGE_SIZE); + for (const { id } of deletions) { + try { await maintenance.deleteObjects({ id, apply: true, signal }); } + catch (error) { + signal.throwIfAborted(); + await db.update(skillPackageDeletions).set({ deleteAfter: new Date(Date.now() + RETRY_MS) }).where(eq(skillPackageDeletions.id, id)); + logger.warn({ deletionId: id, failure: failureDetails(error) }, "Skill package object deletion deferred"); + } + } + } + async function deadline() { + if (!skillPackageStorageEnabled()) return null; + const [row] = await db.execute<{ at: string | Date | null }>(sql`select min(at) as at from ( + select min(migration_at) as at from company_skill_repository_snapshots + union all select min(migration_at) from company_skill_versions + union all select min(maintenance_at) from company_skill_repository_snapshots + union all select min(delete_after) from skill_package_deletions where provider = ${backend().provider} + ) deadlines`); + return row?.at ? new Date(row.at).getTime() : null; + } + function start(coordinator: Pick, "register">) { + let requested = 0, completed = -1, next: number | null = null; + let recovered = false; + return coordinator.register(DELIVERY_QUEUES.skillPackages, { + retryMs: 5_000, pausePassiveMaintenance: true, + onSettled: () => { requested++; }, + async run(signal) { + const generation = requested; + if (!recovered && skillPackageStorageEnabled()) { + // A package previously protected by a run/pin may now be unused. + // Reconsider it once on restart, not on an idle polling interval. + await db.update(snapshots).set({ maintenanceAt: new Date() }).where(and(isNull(snapshots.maintenanceAt), + sql`${snapshots.storage} is not null and ${snapshots.restore} is not null`, + sql`${snapshots.storage}->>'durable' is distinct from 'true'`, + lte(snapshots.createdAt, new Date(Date.now() - SKILL_PACKAGE_RETENTION_MS)))); + recovered = true; + } + await tick(signal); + next = await deadline(); + if (next !== null) next = Math.max(Date.now() + 1_000, next); + completed = generation; + }, + hasPending: async () => false, + nextRunAt: () => requested !== completed ? Date.now() + 5_000 : next, + }); + } + return { tick, deadline, start }; +} diff --git a/server/src/services/skill-snapshot.ts b/server/src/services/skill-snapshot.ts index feb70c27bf0..482bfa9f68a 100644 --- a/server/src/services/skill-snapshot.ts +++ b/server/src/services/skill-snapshot.ts @@ -1,6 +1,9 @@ import { createHash } from 'node:crypto'; import type { CompanySkillVersionFileInventoryEntry } from '@paperclipai/shared'; +export const MAX_SKILL_SNAPSHOT_BYTES = 100 * 1024 * 1024; +export const MAX_SKILL_SNAPSHOT_FILES = 10_000; + export function skillFileBytes(file: { content: string; encoding?: string }): Buffer { return Buffer.from(file.content, file.encoding === 'base64' ? 'base64' : 'utf8'); } diff --git a/server/src/services/skill-source-sync.ts b/server/src/services/skill-source-sync.ts index ae31b25145b..7bba472a7de 100644 --- a/server/src/services/skill-source-sync.ts +++ b/server/src/services/skill-source-sync.ts @@ -103,7 +103,7 @@ function setupActor(setup: Setup): AuthorizationActor { onBehalfOfUserId: setup.refreshUserId === 'local-board' ? null : setup.refreshUserId, source: 'agent_key' } : { type: 'board', userId: setup.refreshUserId, source: setup.refreshLocalTrusted ? 'local_implicit' : 'session', companyIds: [setup.companyId] }; } -async function setupContext(db: Db, setup: Setup): Promise { +export async function skillSourceSetupContext(db: Db, setup: Setup): Promise { if (!setup.refreshUserId && !setup.refreshLocalTrusted) throw forbidden('Choose a GitHub connection to enable automatic updates.'); if (!setup.refreshLocalTrusted) { const [member] = await db.select({ id: companyMemberships.id }).from(companyMemberships) @@ -152,7 +152,7 @@ async function setupContext(db: Db, setup: Setup): Promise { } export function skillSourceSyncService(db: Db, options: { context?: (setup: Setup) => Promise } = {}) { - const contextFor = options.context ?? (setup => setupContext(db, setup)); + const contextFor = options.context ?? (setup => skillSourceSetupContext(db, setup)); const sources = skillSourceService(db); let running: Promise<{ imported: number; checked: number }> | null = null; const shutdown = new AbortController(); @@ -211,7 +211,7 @@ export function skillSourceSyncService(db: Db, options: { context?: (setup: Setu try { const result = await sources.create(binding.companyId, { repositoryUrl: scan.repositoryUrl, trackingRef: binding.trackingRef, commitSha: scan.commitSha, packageMode: scan.packageMode, selectedPaths: scan.candidates.map(candidate => candidate.path), - connectionId: scan.connectionId ?? binding.connectionId }, { ...context, projectBindingId: binding.id, projectBindingLeaseToken: token }, scan); + connectionId: scan.connectionId ?? binding.connectionId }, { ...context, signal: shutdown.signal, projectBindingId: binding.id, projectBindingLeaseToken: token }, scan); sourceId = result.source.id; } catch (error) { if (!(error instanceof Error && 'status' in error && error.status === 409)) throw error; diff --git a/server/src/services/skill-sources.ts b/server/src/services/skill-sources.ts index 7129d97052f..54ed9473116 100644 --- a/server/src/services/skill-sources.ts +++ b/server/src/services/skill-sources.ts @@ -1,3 +1,5 @@ +import { skillPackageManifest } from "./skill-package-files.js"; +import { skillPackageStorageEnabled, skillPackageStore } from "./skill-package-store.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from './delivery-work-notifications.js'; import { reserveAutomaticSkillWork, settleAutomaticSkillWork } from './skill-source-refresh-budget.js'; import { skillSourceRepairAction } from './skill-source-problems.js'; @@ -8,7 +10,7 @@ import path from 'node:path'; import { promises as fs } from 'node:fs'; import { folderService } from './folders.js'; import { and, eq, isNull, lt, or, sql } from 'drizzle-orm'; -import { companySkillSources as sources, companySkillSourceEntries as entries, companySkills, companySkillRepositorySnapshots, projectSkillSources, projects, type Db, authUsers } from '@paperclipai/db'; +import { companySkillSources as sources, companySkillSourceEntries as entries, companySkills, companySkillRepositorySnapshots, projectSkillSources, projects, type Db, type SkillPackageRestoreRecipe, authUsers } from '@paperclipai/db'; import type { CompanySkill, SkillSource, SkillSourceCreateRequest, SkillSourceDiscoveryRequest, SkillSourceSelectionRequest, SkillSourceRefreshResult, SkillSourcePreviewRequest, SkillSourceSettingsRequest, SkillSourceDisconnectRequest, SkillSourceDisconnectResult, SkillSourceBranchesQuery } from '@paperclipai/shared'; import { normalizeAgentUrlKey } from '@paperclipai/shared'; import { conflict, notFound, unprocessable } from '../errors.js'; @@ -17,9 +19,11 @@ import { scanGitHubSkillsWithPackageFallback as scanGitHubSkills, previewGitHubS import { skillAccessService } from './skill-access.js'; import { skillSnapshotHash } from './skill-snapshot.js'; +type PreparedPackages = Map; type Tx = Parameters[0]>[0]; type SourceRow = typeof sources.$inferSelect; export type SkillSourceContext = { + signal?: AbortSignal; actor: { type: 'user' | 'agent' | 'system'; userId?: string | null; agentId?: string | null }; setup?: { userId: string | null; agentId?: string | null; localTrusted: boolean; grantId?: string | null }; projectBindingId?: string; @@ -33,6 +37,29 @@ const scope = (companyId: string, id: string) => and(eq(sources.companyId, compa export function skillSourceService(db: Db) { const skills = companySkillService(db); + const packages = skillPackageStore(db); + async function preparePackages(companyId: string, scan: ScannedSkillSource, selected: string[], sourceId: string, signal?: AbortSignal): Promise { + const prepared: PreparedPackages = new Map(); + if (!skillPackageStorageEnabled()) return prepared; + const candidates = scan.skills.filter(candidate => selected.includes(candidate.path) && !candidate.error); + if (!candidates.length) return prepared; + const prepare = async (files: ScannedSkillSource['skills'][number]['files'], skillPath: string | null) => { + const restore: SkillPackageRestoreRecipe = { version: 1, sourceId, repositoryId: scan.repositoryId, + repositoryUrl: scan.repositoryUrl, commitSha: scan.commitSha, skillPath, manifest: skillPackageManifest(files) }; + const id = await packages.prepare(companyId, files, true, signal, restore); + // Deduplication may preserve another version's kind/encoding metadata. + // Recovery uses the manifest of the actual stored archive. + return { id, restore: { ...restore, manifest: await packages.manifest(companyId, id) } }; + }; + if (scan.packageMode === 'repository') { + if (scan.repositoryPackage?.error || !scan.repositoryFiles?.length) throw unprocessable(scan.repositoryPackage?.error ?? 'Repository package has no files.'); + const saved = await prepare(scan.repositoryFiles, null); + for (const candidate of candidates) prepared.set(candidate.path, saved); + } else { + for (const candidate of candidates) prepared.set(candidate.path, await prepare(candidate.files, candidate.path)); + } + return prepared; + } async function row(companyId: string, id: string, database: Db | Tx = db) { const [source] = await database.select().from(sources).where(scope(companyId, id)); if (!source || source.removedAt) throw notFound('Skill source not found.'); @@ -98,20 +125,23 @@ export function skillSourceService(db: Db) { const identity = createHash('sha256').update(`${scan.repositoryId}:${sourceId}${revision ? `:${revision}` : ''}:${skillPath}`).digest('hex').slice(0, 16); return `github/${scan.repositoryId}/${identity}/${normalizeAgentUrlKey(name) || 'skill'}`; } - async function publish(tx: Tx, source: SourceRow, scan: ScannedSkillSource, selectedPaths: string[], excludedFolders: string[], context: SkillSourceContext, selectionReviewed: boolean): Promise> { + async function publish(tx: Tx, source: SourceRow, scan: ScannedSkillSource, selectedPaths: string[], excludedFolders: string[], context: SkillSourceContext, selectionReviewed: boolean, prepared: PreparedPackages): Promise> { await tx.execute(sql`SET CONSTRAINTS paperclip_skill_runtime_changed DEFERRED`); const mode = scan.packageMode ?? 'skills'; if (mode === 'repository' && scan.skills.length > 0 && (scan.repositoryPackage?.error || !scan.repositoryFiles?.length)) { throw unprocessable(scan.repositoryPackage?.error ?? 'Repository package has no files. Scan it again before importing.'); } - let repositorySnapshotId: string | undefined; - const repositoryHash = mode === 'repository' && scan.repositoryFiles?.length && !scan.repositoryPackage?.error ? skillSnapshotHash(scan.repositoryFiles) : null; - if (repositoryHash) { - await tx.insert(companySkillRepositorySnapshots).values({ companyId: source.companyId, contentHash: repositoryHash, fileInventory: scan.repositoryFiles! }) - .onConflictDoNothing(); - const [snapshot] = await tx.select({ id: companySkillRepositorySnapshots.id }).from(companySkillRepositorySnapshots).where(and( - eq(companySkillRepositorySnapshots.companyId, source.companyId), eq(companySkillRepositorySnapshots.contentHash, repositoryHash))); - repositorySnapshotId = snapshot!.id; + let repositorySnapshotId = mode === 'repository' ? prepared.values().next().value?.id : undefined; + const hasSelected = scan.skills.some(candidate => selectedPaths.includes(candidate.path) && !candidate.error); + const repositoryHash = mode === 'repository' && hasSelected ? skillSnapshotHash(scan.repositoryFiles!) : null; + if (repositoryHash && !repositorySnapshotId) { + // Compatibility writer for a rolling deployment. Never save an otherwise + // unused repository when no valid skill is selected. + const [saved] = await tx.insert(companySkillRepositorySnapshots).values({ companyId: source.companyId, + contentHash: repositoryHash, fileInventory: scan.repositoryFiles! }).onConflictDoNothing().returning({ id: companySkillRepositorySnapshots.id }); + repositorySnapshotId = saved?.id ?? (await tx.select({ id: companySkillRepositorySnapshots.id }) + .from(companySkillRepositorySnapshots).where(and(eq(companySkillRepositorySnapshots.companyId, source.companyId), + eq(companySkillRepositorySnapshots.contentHash, repositoryHash))))[0]!.id; } const previous = await tx.select().from(entries).where(and(eq(entries.companyId, source.companyId), eq(entries.sourceId, source.id))); const imported: CompanySkill[] = [], updated: CompanySkill[] = []; @@ -157,6 +187,9 @@ export function skillSourceService(db: Db) { if (changed) { const version = await skills.createVersion(source.companyId, skillId!, { label: `GitHub ${scan.commitSha.slice(0, 8)}` }, context.actor, { database: tx, skipInventoryRefresh: true, skill: installed, fileInventory: candidate.files, + packageSnapshotId: prepared.get(candidate.path)?.id, + packageFileMap: repositorySnapshotId && prepared.has(candidate.path) ? candidate.files.map(file => ({ path: file.path, kind: file.kind, + sourcePath: file.path === 'SKILL.md' ? candidate.path : path.posix.join(path.posix.dirname(candidate.path), file.path) })) : null, ...(repositorySnapshotId ? { repositorySnapshotId, repositorySkillPath: candidate.path } : {}), }); installed.currentVersionId = version.id; @@ -182,6 +215,10 @@ export function skillSourceService(db: Db) { if (current) await tx.update(companySkills).set({ metadata: { ...current.metadata, skillSourceId: source.id, skillSourcePath: old.path, skillSourceState: 'removed' } }).where(and(eq(companySkills.companyId, source.companyId), eq(companySkills.id, old.skillId))); } } + const publicationStore = skillPackageStore(tx as unknown as Db); + for (const { id, restore } of new Map([...prepared.values()].map(saved => [saved.id, saved])).values()) { + await publicationStore.bindRestore(source.companyId, id, restore); + } // HEAD remains the default-branch alias across refreshes and branch renames. await tx.update(sources).set({ repositoryId: scan.repositoryId, repositoryUrl: scan.repositoryUrl, fullName: scan.fullName, packageMode: mode, repositoryPackage: scan.repositoryPackage ?? null, @@ -203,8 +240,11 @@ export function skillSourceService(db: Db) { await context.authorize('skills.import', { sourceType: 'git', sourceLocator: scan.repositoryUrl, skillKey: newSkillKey(scan, candidate.path, candidate.name, id) }); } + const prepared = await preparePackages(companyId, scan, input.selectedPaths, id, context.signal); + context.signal?.throwIfAborted(); const published = await db.transaction(async tx => { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); const [source] = await tx.insert(sources).values({ id, companyId, repositoryId: scan.repositoryId, repositoryUrl: scan.repositoryUrl, fullName: scan.fullName, packageMode: scan.packageMode ?? 'skills', trackingRef: context.projectBindingId && input.trackingRef === 'HEAD' ? 'HEAD' : scan.trackingRef, connectionId: scan.connectionId ?? input.connectionId ?? null, lastAttemptAt: new Date(), accessScope: context.projectBindingId ? 'projects' : 'company', importNewSkills: Boolean(context.projectBindingId), @@ -215,7 +255,7 @@ export function skillSourceService(db: Db) { if (!binding?.active || (context.projectBindingLeaseToken && binding.leaseToken !== context.projectBindingLeaseToken)) throw conflict('Project repository changed during import.'); await tx.update(projectSkillSources).set({ sourceId: source.id, status: 'ready', lastError: null }).where(eq(projectSkillSources.id, binding.id)); } - const result = await publish(tx, source, scan, input.selectedPaths, input.excludedFolders ?? [], context, true); + const result = await publish(tx, source, scan, input.selectedPaths, input.excludedFolders ?? [], context, true, prepared); return { ...result, source: await detail(companyId, id, tx) }; }); context.publishCommittedActivity?.(); @@ -289,11 +329,14 @@ export function skillSourceService(db: Db) { } } await authorizeScan(source, scan, selectedPaths, context); + const prepared = await preparePackages(companyId, scan, selectedPaths, id, options.signal ?? context.signal); + (options.signal ?? context.signal)?.throwIfAborted(); const published = await db.transaction(async tx => { if (!options.automatic) await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); const [locked] = await tx.select().from(sources).where(scope(companyId, id)).for('update'); if (!locked || locked.leaseToken !== token || locked.revision !== source.revision || locked.leaseExpiresAt! < new Date()) throw conflict('Source refresh expired or changed. Try again.'); - const result = await publish(tx, locked, scan, selectedPaths, selection?.excludedFolders ?? source.excludedFolders, context, Boolean(selection)); + const result = await publish(tx, locked, scan, selectedPaths, selection?.excludedFolders ?? source.excludedFolders, context, Boolean(selection), prepared); if (selection || scan.connectionId !== undefined) await tx.update(sources).set({ ...(selection ? { enabled: true } : {}), connectionId: scan.connectionId ?? connectionId, ...(source.refreshGrantId === null && reader.grantId ? { refreshGrantId: reader.grantId } : {}), ...(selection?.connectionId !== undefined || (!source.refreshUserId && !source.refreshLocalTrusted && context.setup) ? { refreshUserId: context.setup?.userId, refreshAgentId: context.setup?.agentId ?? null, refreshLocalTrusted: context.setup?.localTrusted ?? false, refreshGrantId: reader.grantId ?? null } : {}) }).where(scope(companyId, id)); @@ -383,6 +426,7 @@ export function skillSourceService(db: Db) { } async function remove(tx: Tx, copiedCount: number, deletedCount: number) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); // Stop both pending attachments and an in-flight refresh before removing tracking. await tx.update(projectSkillSources).set({ active: false, leaseToken: null, leaseExpiresAt: null }) .where(and(eq(projectSkillSources.companyId, companyId), eq(projectSkillSources.sourceId, id))); diff --git a/server/src/services/skill-version-cache.ts b/server/src/services/skill-version-cache.ts index 9b243607806..e81e253314d 100644 --- a/server/src/services/skill-version-cache.ts +++ b/server/src/services/skill-version-cache.ts @@ -1,9 +1,10 @@ +import { skillPackageStore, skillVersionMetadataColumns } from "./skill-package-store.js"; import { AsyncLocalStorage } from "node:async_hooks"; import { randomUUID } from "node:crypto"; import { constants, promises as fs } from "node:fs"; import path from "node:path"; import { and, eq, sql } from "drizzle-orm"; -import { companySkillVersions, companySkillRepositorySnapshots, heartbeatRuns, +import { companySkillVersions, heartbeatRuns, nativeRunFinalizations, environmentLeases, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { resolvePaperclipInstanceRoot } from "../home-paths.js"; import { assertSkillSnapshotPath } from "./skill-snapshot.js"; @@ -17,11 +18,6 @@ export const SKILL_VERSION_CACHE_IDLE_MS = 60 * 60 * 1000; const locks = new AsyncLocalStorage>(); const uuid = /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i; export class SkillCacheBusy extends Error {} -// Repository snapshots are immutable and referenced with restrictive foreign -// keys. Cache only validated entrypoint names, never large file contents, so an -// hourly sweep does not transfer the same repository for each skill/version. -const verifiedSnapshots = new WeakMap>>(); - /** Hold through publication AND durable run pinning, not just file creation. */ export async function withSkillVersionCacheLock(db: Db, companyId: string, action: () => Promise, tryOnly = false): Promise { const held = locks.getStore() ?? []; @@ -97,6 +93,23 @@ export async function lastUsedAt(filename: string): Promise { } } +export function skillRunNeedsFilesSql() { + return sql`( + ${heartbeatRuns.status} not in ('succeeded', 'failed', 'timed_out', 'cancelled', 'interrupted') + or ${heartbeatRuns.finishedAt} is null + or ${heartbeatRuns.controllerLeaseExpiresAt} > now() + or exists (select 1 from ${nativeRunFinalizations} f + where f.company_id = ${heartbeatRuns.companyId} and f.run_id = ${heartbeatRuns.id} + and (f.phase <> 'committed' or f.lease_owner is not null or f.recovery_state is not null)) + or exists (select 1 from ${environmentLeases} l + where l.company_id = ${heartbeatRuns.companyId} and l.heartbeat_run_id = ${heartbeatRuns.id} + and (l.status = 'pending_cleanup' or l.cleanup_status = 'failed' + or l.metadata ? 'nativeWorkspaceExportResume' + or (l.released_at is null and not coalesce(l.status = 'retained' + and l.lease_policy = 'reuse_by_environment' and l.cleanup_status = 'success', false)))) + )`; +} + // A terminal run can still own provider execution or recovery. Only fully settled // runs release their pins; resume preparation reacquires the same cache lock. export async function protectedVersions(db: Db, companyId: string): Promise<{ versions: Set | null; leaseDeadlines: Map }> { @@ -111,20 +124,7 @@ export async function protectedVersions(db: Db, companyId: string): Promise<{ ve and not coalesce(${heartbeatRuns.contextSnapshot} ? 'skillVersions', false) then ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput' end`, }) - .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), sql`( - ${heartbeatRuns.status} not in ('succeeded', 'failed', 'timed_out', 'cancelled', 'interrupted') - or ${heartbeatRuns.finishedAt} is null - or ${heartbeatRuns.controllerLeaseExpiresAt} > now() - or exists (select 1 from ${nativeRunFinalizations} f - where f.company_id = ${heartbeatRuns.companyId} and f.run_id = ${heartbeatRuns.id} - and (f.phase <> 'committed' or f.lease_owner is not null or f.recovery_state is not null)) - or exists (select 1 from ${environmentLeases} l - where l.company_id = ${heartbeatRuns.companyId} and l.heartbeat_run_id = ${heartbeatRuns.id} - and (l.status = 'pending_cleanup' or l.cleanup_status = 'failed' - or l.metadata ? 'nativeWorkspaceExportResume' - or (l.released_at is null and not coalesce(l.status = 'retained' - and l.lease_policy = 'reuse_by_environment' and l.cleanup_status = 'success', false)))) - )`)); + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), skillRunNeedsFilesSql())); const versions = new Set(); const leaseDeadlines = new Map(); const unknown = () => ({ versions: null, leaseDeadlines }); @@ -174,27 +174,28 @@ function validateInventory(inventory: CompanySkillVersionFileInventoryEntry[]) { } export async function hasRestorableSnapshot(db: Db, companyId: string, versionId: string) { - const [version] = await db.select().from(companySkillVersions).where(and( + const [version] = await db.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.id, versionId))); if (!version) return false; - if (version.repositorySnapshotId) { - if (!version.repositorySkillPath) return false; - let cache = verifiedSnapshots.get(db); - if (!cache) { cache = new Map(); verifiedSnapshots.set(db, cache); } - const key = `${companyId}:${version.repositorySnapshotId}`; - const verified = cache.get(key); - if (verified) return verified.has(version.repositorySkillPath); - const [snapshot] = await db.select({ inventory: companySkillRepositorySnapshots.fileInventory }) - .from(companySkillRepositorySnapshots).where(and(eq(companySkillRepositorySnapshots.companyId, companyId), - eq(companySkillRepositorySnapshots.id, version.repositorySnapshotId))); - if (!snapshot || !validateInventory(snapshot.inventory)) return false; - const entrypoints = new Set(snapshot.inventory.filter(file => file.encoding !== "base64" - && repositorySkillFrontmatter(file.content)).map(file => file.path)); - if (cache.size >= 128) cache.delete(cache.keys().next().value!); - cache.set(key, entrypoints); - return entrypoints.has(version.repositorySkillPath); + const packageId = version.repositorySnapshotId ?? version.packageSnapshotId; + const store = skillPackageStore(db); + if (packageId) { + if (version.repositorySnapshotId && !version.repositorySkillPath) return false; + const manifest = await store.manifest(companyId, packageId); + const entrypoint = version.repositorySkillPath ?? version.packageFileMap?.find(file => file.path === "SKILL.md")?.sourcePath ?? "SKILL.md"; + const entry = manifest.files.find(file => file.path === entrypoint); + if (!entry) return false; + // Git packages intentionally permit byte eviction. Their immutable recipe, + // not a network probe during cleanup, is the restoration contract. + if ((await store.descriptor(companyId, packageId)).restorable) return true; + if (version.repositorySnapshotId && entry.encoding === "base64") return false; + // Validate original frontmatter inside the same isolated storage verification, + // outside the preparation lock and without rebuilding the shared cache. + return store.verifyDurable(companyId, packageId, undefined, version.repositorySnapshotId + ? async root => Boolean(repositorySkillFrontmatter(await fs.readFile(path.join(root, entrypoint), "utf8"))) : undefined); } - return validateInventory(version.fileInventory) && version.fileInventory.some(file => file.path === "SKILL.md"); + const files = await store.versionFiles(version); + return validateInventory(files) && files.some(file => file.path === "SKILL.md"); } // Counts allocated bytes without following links, matching what df can reclaim. diff --git a/server/src/services/skill-version-retention.ts b/server/src/services/skill-version-retention.ts index f959efa4ef4..476ccb011ac 100644 --- a/server/src/services/skill-version-retention.ts +++ b/server/src/services/skill-version-retention.ts @@ -1,3 +1,4 @@ +import { withSkillPackageReadScope } from "./skill-package-store.js"; import { randomUUID } from "node:crypto"; import { promises as fs } from "node:fs"; import path from "node:path"; @@ -439,7 +440,7 @@ export function skillVersionRetentionService(db: Db, defaults: SkillVersionPrune async function pruneCompany(companyId: string, options: SkillVersionPruneOptions & { cursor?: CompanyCursor | null } = {}) { if (scanning.has(companyId)) return { ...emptyResult(), checked: 1, nextRunAt: Date.now() + 30_000, skipped: { scan_busy: 1 } }; scanning.add(companyId); - try { return await pruneCompanyPage(companyId, options); } + try { return await withSkillPackageReadScope(db, () => pruneCompanyPage(companyId, options)); } finally { scanning.delete(companyId); } } diff --git a/server/src/storage/failure-cleanup.ts b/server/src/storage/failure-cleanup.ts new file mode 100644 index 00000000000..c4fa2ea70c1 --- /dev/null +++ b/server/src/storage/failure-cleanup.ts @@ -0,0 +1,17 @@ +export const STORAGE_FAILURE_CLEANUP_MS = 5_000; + +export async function cleanupStorageFailure(cleanup: (signal: AbortSignal) => Promise): Promise { + const controller = new AbortController(); + let timer: ReturnType | undefined; + try { + await Promise.race([ + Promise.resolve().then(() => cleanup(controller.signal)), + new Promise(resolve => { + timer = setTimeout(() => { controller.abort(); resolve(); }, STORAGE_FAILURE_CLEANUP_MS); + }), + ]); + } catch { + } finally { + clearTimeout(timer); + } +} diff --git a/server/src/storage/local-disk-provider.ts b/server/src/storage/local-disk-provider.ts index df3ed649fa6..9ee950117f3 100644 --- a/server/src/storage/local-disk-provider.ts +++ b/server/src/storage/local-disk-provider.ts @@ -48,7 +48,7 @@ export function createLocalDiskStorageProvider(baseDir: string): StorageProvider const tempPath = `${targetPath}.tmp-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; try { - await fs.writeFile(tempPath, input.body); + await fs.writeFile(tempPath, input.body, { signal: input.signal }); await fs.rename(tempPath, targetPath); } finally { await fs.rm(tempPath, { force: true }); } }, @@ -66,7 +66,7 @@ export function createLocalDiskStorageProvider(baseDir: string): StorageProvider ? input.range.end - input.range.start + 1 : stat.size; return { - stream: createReadStream(filePath, streamOptions), + stream: createReadStream(filePath, { ...streamOptions, signal: input.signal }), contentLength, lastModified: stat.mtime, }; diff --git a/server/src/storage/s3-multipart.ts b/server/src/storage/s3-multipart.ts index 2b8e0d951d4..67db0d7c330 100644 --- a/server/src/storage/s3-multipart.ts +++ b/server/src/storage/s3-multipart.ts @@ -2,6 +2,7 @@ import { AbortMultipartUploadCommand, CompleteMultipartUploadCommand, CreateMult import { Readable } from "node:stream"; import type { PutObjectInput } from "./types.js"; import { withS3FailureDiagnostics } from "./s3-failure.js"; +import { cleanupStorageFailure } from "./failure-cleanup.js"; /** Each part is streamed too: multipart must not become another whole-file * buffer or a fixed part-count ceiling below the storage provider's limits. */ @@ -13,7 +14,7 @@ export async function putS3Multipart(client: S3Client, bucket: string, key: stri // Scale part sizes to the S3 part-count limit while retaining stream backpressure. const partSize = Math.max(8 * 1024 * 1024, Math.ceil(input.contentLength / 10_000)); try { - uploadId = (await withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType })))).UploadId; + uploadId = (await withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: input.signal }))).UploadId; if (!uploadId) throw new Error("S3 did not return a multipart upload id"); const parts: { ETag: string; PartNumber: number }[] = []; for (let offset = 0; offset < input.contentLength;) { @@ -35,19 +36,19 @@ export async function putS3Multipart(client: S3Client, bucket: string, key: stri } })()); try { - const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: parts.length + 1, Body: body, ContentLength: length }))); + const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: parts.length + 1, Body: body, ContentLength: length }), { abortSignal: input.signal })); if (remaining || !result.ETag) throw new Error("S3 upload part is incomplete"); parts.push({ ETag: result.ETag, PartNumber: parts.length + 1 }); } finally { body.destroy(); } offset += length; } if (pending.length || !(await iterator.next()).done) throw new Error("S3 upload source exceeds its declared size"); - await withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }))); + await withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }), { abortSignal: input.signal })); } catch (error) { - if (uploadId) await client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })).catch(() => {}); + if (uploadId) await cleanupStorageFailure(signal => client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId }), { abortSignal: signal })); throw error; } finally { source.destroy(); - await iterator.return?.(); + await cleanupStorageFailure(async () => iterator.return?.()); } } diff --git a/server/src/storage/s3-provider.ts b/server/src/storage/s3-provider.ts index b96ff003be8..faa2deb949f 100644 --- a/server/src/storage/s3-provider.ts +++ b/server/src/storage/s3-provider.ts @@ -95,6 +95,7 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid ContentType: input.contentType, ContentLength: input.contentLength, }), + { abortSignal: input.signal }, )); }, @@ -158,6 +159,7 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid Bucket: bucket, Key: key, }), + { abortSignal: input.signal }, )); }, }; diff --git a/server/src/storage/service.ts b/server/src/storage/service.ts index b0020308556..a78752993e5 100644 --- a/server/src/storage/service.ts +++ b/server/src/storage/service.ts @@ -3,9 +3,22 @@ import path from "node:path"; import { Readable } from "node:stream"; import type { StorageService, StorageProvider, PutFileInput, PutFileResult } from "./types.js"; import { badRequest, forbidden, unprocessable } from "../errors.js"; +import { cleanupStorageFailure } from "./failure-cleanup.js"; const MAX_SEGMENT_LENGTH = 120; +async function cancellable(signal: AbortSignal | undefined, action: () => Promise): Promise { + signal?.throwIfAborted(); + if (!signal) return action(); + let abort: () => void = () => {}; + const cancelled = new Promise((_resolve, reject) => { + abort = () => reject(signal.reason); + signal.addEventListener("abort", abort, { once: true }); + }); + try { return await Promise.race([action(), cancelled]); } + finally { signal.removeEventListener("abort", abort); } +} + function sanitizeSegment(value: string): string { const cleaned = value .trim() @@ -107,9 +120,10 @@ export function createStorageService(provider: StorageProvider): StorageService body: input.body, contentType, contentLength: byteSize, + signal: input.signal, }); } catch (error) { - await provider.deleteObject({ objectKey }).catch(() => {}); + await cleanupStorageFailure(signal => provider.deleteObject({ objectKey, signal })); throw error; } finally { if (input.body instanceof Readable) input.body.destroy(); @@ -127,17 +141,17 @@ export function createStorageService(provider: StorageProvider): StorageService async getObject(companyId: string, objectKey: string, options) { ensureCompanyPrefix(companyId, objectKey); - return provider.getObject({ objectKey, range: options?.range }); + return provider.getObject({ objectKey, range: options?.range, signal: options?.signal }); }, - async headObject(companyId: string, objectKey: string) { + async headObject(companyId: string, objectKey: string, options) { ensureCompanyPrefix(companyId, objectKey); - return provider.headObject({ objectKey }); + return cancellable(options?.signal, () => provider.headObject({ objectKey, signal: options?.signal })); }, - async deleteObject(companyId: string, objectKey: string) { + async deleteObject(companyId: string, objectKey: string, options) { ensureCompanyPrefix(companyId, objectKey); - await provider.deleteObject({ objectKey }); + await cancellable(options?.signal, () => provider.deleteObject({ objectKey, signal: options?.signal })); }, }; } diff --git a/server/src/storage/types.ts b/server/src/storage/types.ts index 36d941147f8..60f6d1d24a6 100644 --- a/server/src/storage/types.ts +++ b/server/src/storage/types.ts @@ -3,6 +3,7 @@ import type { Readable } from "node:stream"; export interface PutObjectInput { objectKey: string; + signal?: AbortSignal; // Readable bodies stream straight to the backend (contentLength must be the // exact byte size); Buffer stays supported for small payloads. body: Buffer | Readable; @@ -45,6 +46,7 @@ export interface StorageProvider { } export type PutFileInput = { + signal?: AbortSignal; /** Server-allocated, company-prefixed key for durable idempotent uploads. Never accept from client input. */ objectKey?: string; companyId: string; @@ -65,7 +67,7 @@ export interface PutFileResult { export interface StorageService { provider: StorageProviderId; putFile(input: PutFileInput): Promise; - getObject(companyId: string, objectKey: string, options?: Pick): Promise; - headObject(companyId: string, objectKey: string): Promise; - deleteObject(companyId: string, objectKey: string): Promise; + getObject(companyId: string, objectKey: string, options?: Pick): Promise; + headObject(companyId: string, objectKey: string, options?: Pick): Promise; + deleteObject(companyId: string, objectKey: string, options?: Pick): Promise; } diff --git a/ui/src/api/companySkills.ts b/ui/src/api/companySkills.ts index 726d7eb9585..9b671d57038 100644 --- a/ui/src/api/companySkills.ts +++ b/ui/src/api/companySkills.ts @@ -38,6 +38,7 @@ import type { CompanySkillUpdateRequest, CompanySkillUpdateStatus, CompanySkillVersion, + CompanySkillVersionSummary, CompanySkillVersionCreateRequest, } from "@paperclipai/shared"; import { api } from "./client"; @@ -66,7 +67,7 @@ export const companySkillsApi = { `/companies/${encodeURIComponent(companyId)}/skills/${encodeURIComponent(skillId)}`, ), versions: (companyId: string, skillId: string) => - api.get( + api.get( `/companies/${encodeURIComponent(companyId)}/skills/${encodeURIComponent(skillId)}/versions`, ), version: (companyId: string, skillId: string, versionId: string) => diff --git a/ui/src/components/skill-studio/AgentsUsingSkillDialog.tsx b/ui/src/components/skill-studio/AgentsUsingSkillDialog.tsx index 439e0ce92c1..c4e747ae4db 100644 --- a/ui/src/components/skill-studio/AgentsUsingSkillDialog.tsx +++ b/ui/src/components/skill-studio/AgentsUsingSkillDialog.tsx @@ -6,7 +6,7 @@ import type { AgentDesiredSkillEntry, CompanySkillDetail, CompanySkillUsageAgent, - CompanySkillVersion, + CompanySkillVersionSummary, } from "@paperclipai/shared"; import { Link } from "@/lib/router"; import { agentsApi } from "@/api/agents"; @@ -130,7 +130,7 @@ export function AgentsUsingSkillDialog({ if (!open) setConfirmRemoveId(null); }, [open]); - const versions = useMemo( + const versions = useMemo( () => [...(versionsQuery.data ?? [])].sort((a, b) => b.revisionNumber - a.revisionNumber), [versionsQuery.data], ); @@ -308,7 +308,7 @@ function AgentRow({ icon: string | null; canManage: boolean; hasVersions: boolean; - versions: CompanySkillVersion[]; + versions: CompanySkillVersionSummary[]; latestRevision: number | null; pinnedRevision: number | null; busy: boolean; diff --git a/ui/src/pages/CompanySkills.production.tsx b/ui/src/pages/CompanySkills.production.tsx index c0b9ab9a129..c01590d10f2 100644 --- a/ui/src/pages/CompanySkills.production.tsx +++ b/ui/src/pages/CompanySkills.production.tsx @@ -23,7 +23,7 @@ import type { CompanySkillTrustLevel, CompanySkillUpdateRequest, CompanySkillUpdateStatus, - CompanySkillVersion, + CompanySkillVersionSummary, } from "@paperclipai/shared"; import { companySkillsApi } from "../api/companySkills"; import { foldersApi } from "../api/folders"; @@ -2277,7 +2277,7 @@ function AttachAgentsPopover({ }: { agents: AttachAgentOption[]; attachedAgentIds: string[]; - versions: CompanySkillVersion[]; + versions: CompanySkillVersionSummary[]; selectedVersionId: string | null; pending: boolean; onSubmit: (nextIds: string[], versionId: string | null) => void; @@ -2571,12 +2571,12 @@ function currentVersionSelection(detail: CompanySkillDetail | null | undefined) return selected ?? null; } -function versionLabel(version: CompanySkillVersion | null | undefined) { +function versionLabel(version: CompanySkillVersionSummary | null | undefined) { if (!version) return "Latest"; return `v${version.revisionNumber}${version.label ? ` · ${version.label}` : ""}`; } -export function getSkillVersionDiffSelection(versions: CompanySkillVersion[], targetVersionId?: string | null) { +export function getSkillVersionDiffSelection(versions: CompanySkillVersionSummary[], targetVersionId?: string | null) { const sorted = [...versions].sort((a, b) => b.revisionNumber - a.revisionNumber); const right = targetVersionId ? sorted.find((version) => version.id === targetVersionId) ?? null @@ -2601,15 +2601,27 @@ function SkillVersionDiffDialog({ }: { open: boolean; onOpenChange: (open: boolean) => void; - versions: CompanySkillVersion[]; + versions: CompanySkillVersionSummary[]; leftVersionId: string | null; rightVersionId: string | null; onLeftVersionChange: (id: string | null) => void; onRightVersionChange: (id: string | null) => void; }) { const sorted = [...versions].sort((a, b) => b.revisionNumber - a.revisionNumber); - const left = sorted.find((version) => version.id === leftVersionId) ?? null; - const right = sorted.find((version) => version.id === rightVersionId) ?? null; + const selectedLeft = sorted.find((version) => version.id === leftVersionId); + const selectedRight = sorted.find((version) => version.id === rightVersionId); + const leftQuery = useQuery({ + queryKey: [...queryKeys.companySkills.versions(selectedLeft?.companyId ?? "", selectedLeft?.companySkillId ?? ""), leftVersionId], + queryFn: () => companySkillsApi.version(selectedLeft!.companyId, selectedLeft!.companySkillId, selectedLeft!.id), + enabled: open && Boolean(selectedLeft), + }); + const rightQuery = useQuery({ + queryKey: [...queryKeys.companySkills.versions(selectedRight?.companyId ?? "", selectedRight?.companySkillId ?? ""), rightVersionId], + queryFn: () => companySkillsApi.version(selectedRight!.companyId, selectedRight!.companySkillId, selectedRight!.id), + enabled: open && Boolean(selectedRight), + }); + const left = leftQuery.data ?? null; + const right = rightQuery.data ?? null; const allPaths = useMemo(() => { const paths = new Set(); for (const file of left?.fileInventory ?? []) paths.add(file.path); @@ -2651,6 +2663,8 @@ function SkillVersionDiffDialog({
Diff · skill files + {(leftQuery.error || rightQuery.error) &&

{(leftQuery.error ?? rightQuery.error)?.message}

} + {(leftQuery.isLoading || rightQuery.isLoading) &&

Loading selected versions…

}