diff --git a/doc/DATABASE.md b/doc/DATABASE.md index bf9f4a03333..35838d36da3 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -1252,6 +1252,27 @@ register before writing, or they can remain unseen until the next startup or another notification. Tests should verify both the producer and its outer transaction boundary. +### Installed skill packages + +`company_skill_repository_snapshots` retains its identity and company/content-hash +uniqueness. Despite its historical name, it can also hold editable skill packages. +Migration 0398 adds `storage` (archive/manifest receipts) and `upload` (a bounded +pending attempt). `file_inventory` remains the legacy payload, defaulting to `[]`. +`company_skill_versions.package_snapshot_id` and `package_file_map` preserve the +public version file inventory without storing its contents again. Existing repository +snapshot/entrypoint references and version IDs remain unchanged. Reference indexes +support bounded orphan checks; restrictive FKs protect retained version packages. + +`skill_package_deletions` is a durable object-deletion outbox populated by snapshot +update/delete triggers. Its company ID deliberately has no cascading company FK: +cleanup must retain exact object keys after company deletion. Upload promotion skips +retirement; replacing a pending token retires only the old attempt's unique keys. +Migration 0399 adds exact-commit restore recipes and indexed conversion/maintenance +deadlines. A bounded coordinator worker automatically converts and clears verified +legacy payloads; no migration command is required. Warm reads exclude recipe manifests. +See [package storage](skill-package-storage.md) for acquisition, retention, compatibility +overrides, independent restore verification and coordinated backup policy. + ### Project repository skill synchronization `project_skill_sources` stores durable workspace import jobs and originating project diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 035d338de5c..a301752f7a6 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2885,6 +2885,11 @@ Environment overrides: non-numeric value falls back to the default. - `PAPERCLIP_WORKSPACE_COMPACTION_ENABLED=false` turns compaction off. Any other value keeps it on. +- `PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED=false` disables external writes and automatic + package maintenance for a coordinated rolling upgrade. Default: enabled. Legacy + conversion runs automatically in bounded batches, outside turn preparation. + Existing external packages remain readable with the override. See + [package storage and acquisition](skill-package-storage.md). - `PAPERCLIP_SKILL_VERSION_RETENTION_KEEP=` sets how many of the newest materialized versions of each skill stay on disk (see "Skill Version Snapshot Retention" above). The default is `0`. A negative, empty, or non-numeric value diff --git a/doc/PRODUCT.md b/doc/PRODUCT.md index 8bd41009632..2e2091de5bf 100644 --- a/doc/PRODUCT.md +++ b/doc/PRODUCT.md @@ -198,6 +198,12 @@ Missing references are advisory: a valid selected skill can still be imported. Unavailable files are listed, and existing installed dependencies are retained when a bounded refresh cannot preserve them. File limits and content validation still apply; importing never executes repository code. +Authored and unproven legacy versions retain an independent recovery copy. Unused Git +packages with an exact restore recipe may expire; later use requires access to that +original repository and commit. +External package storage keeps repository contents out of the control-plane database; +per-turn checks reuse verified local files while rechecking current skill access. +See [automatic storage and recovery](skill-package-storage.md). Project-derived skills inherit current project access. Access through any originating project is sufficient, including the responsible-user restrictions on agent execution. diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 3b9d918e848..6c00700be39 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -2362,7 +2362,13 @@ unavailable. Preserve current ownership and newer-work fences. See on their snapshot. Importing does not install dependencies or configure host tools. - Stage and audit complete packages before publishing. Scripts are allowed through the existing content audit and never run on import. Persist binary bytes and executable - flags in immutable version inventories (legacy entries default to UTF-8/non-executable). + flags in immutable package snapshots (legacy entries default to UTF-8/non-executable). + Compatible readers support inline and external inventories. PostgreSQL retains + receipts, file mappings and exact Git restore recipes; the storage backend retains + bytes. No package is persisted without a valid selected skill. Legacy conversion + is automatic and bounded. Unused Git caches can expire and reacquire the recorded + commit after current authorization and hash checks; authored/unproven legacy bytes + remain durable. See [skill package storage](skill-package-storage.md). - Refresh is serialized per source, checks saved selection revisions, and publishes valid skills plus per-entry outcomes with required activity records in one transaction. Failed packages retain prior versions; access/scan/download failures retain all content. @@ -2374,8 +2380,10 @@ unavailable. Preserve current ownership and newer-work fences. See - Runtime version directories are disposable caches. An hourly, bounded sweep may evict non-current copies unused for one hour, preserving active and unresolved execution pins. Selection, materialization, and pin persistence serialize with - eviction across processes. Missing copies rebuild from immutable database bytes - without upstream access. See [recoverable disk cleanup](recoverable-disk-cleanup.md) + eviction across processes. Missing copies rebuild from immutable stored packages + or legacy database bytes without upstream access. Warm preparation uses manifests + and file identities, avoids repository-content SQL/object reads, and does not + reconcile unrelated library entries for known selected repository skills. See [recoverable disk cleanup](recoverable-disk-cleanup.md) for rollout controls, legacy grace periods, and interrupted cleanup recovery. - Manually selected sources retain reviewed selection, with new candidates checked by default and explicit folder exclusions retained. Upstream deletion keeps installed content with diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 8153e21253f..a4d5197cc84 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -445,3 +445,13 @@ timer remains; an empty buffer performs no database work. Existing Muse durable sleep blockers, external worker stop evidence, and ingress requirements remain unchanged. This change establishes database quiet for inactive Muse, not host sleep eligibility for a configured Muse connection. + +## Skill package maintenance + +The package delivery worker automatically converts legacy snapshots and versions. +Unfinished `migration_at` rows block idle sleep until their durable contents are +verified and conversion finishes. Package/object/native-asset retention timestamps +are earliest cleanup eligibility, not promised delivery deadlines; optional cleanup +may wait for the next ingress wake. Workers reconstruct persisted rows and filesystem +access leases on startup, and use the existing coordinator's drain/standby/shutdown +controls. No external Cloud alarm or provider suspension is established by these tests. diff --git a/doc/plans/2026-10-28-skill-package-performance-results.json b/doc/plans/2026-10-28-skill-package-performance-results.json new file mode 100644 index 00000000000..fde86e8206f --- /dev/null +++ b/doc/plans/2026-10-28-skill-package-performance-results.json @@ -0,0 +1,9741 @@ +{ + "schema": "paperclip.skill-package-performance-results.v1", + "date": "2026-10-28", + "fixture": { + "baseFiles": 1349, + "baseJsonBytes": 22799129, + "kind": "synthetic repository; additional entrypoints added for 2- and 10-skill cases" + }, + "notes": [ + "Local synthetic PostgreSQL and filesystem benchmark, with native asset preparation, fresh agent authorization, 155 unrelated skills and 5 ms database delay in each direction. Not deployed Railway latency.", + "Twenty numbered warm samples; warm-new-service is separate. Peak RSS is the entire benchmark process, including application, fixture, and cold allocations, not incremental package memory.", + "Instance disk covers durable local object files plus extracted packages, runtime versions and native assets; filesystem block sharing/compression can affect allocated bytes.", + "Source hashes record the measured code. Subsequent changes bound cold manifest/eviction I/O and serialize version-history hydration; the measured warm runtime algorithm is unchanged." + ], + "measurements": [ + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "3f45986c6dd14f62b04e48b88d699b7c401d1ac877a894f7666696c84df8cc24", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 1, + "instanceDisk": { + "allocatedBytes": 96739328, + "fileBytes": 81013397, + "files": 4055 + }, + "repositoryHash": "5cedce49bfb80e1430df76f4b3889a1b20095545e9aeb575d1b037ab3c6ecdbe", + "repositoryFiles": 1349, + "runtimeFiles": 1350, + "runtimeBytes": 22377362, + "samples": [ + { + "label": "cold", + "elapsedMs": 3714.5453749999997, + "nativeAssetMs": 945.963667, + "sqlResponseBytes": 19994, + "objectGets": 2, + "objectReadBytes": 13450429, + "rssBytes": 839942144, + "cpuUserMs": 2924.79, + "cpuSystemMs": 2069.853, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 6 + }, + { + "label": "warm-1", + "elapsedMs": 270.38466699999844, + "nativeAssetMs": 27.069083000002138, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 866238464, + "cpuUserMs": 60.196, + "cpuSystemMs": 43.843, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 5 + }, + { + "label": "warm-2", + "elapsedMs": 267.416874999999, + "nativeAssetMs": 33.68354199999885, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 870596608, + "cpuUserMs": 56.781, + "cpuSystemMs": 40.212, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 5 + }, + { + "label": "warm-3", + "elapsedMs": 403.48920799999905, + "nativeAssetMs": 122.44841699999961, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 871415808, + "cpuUserMs": 56.459, + "cpuSystemMs": 51.775, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 5 + }, + { + "label": "warm-4", + "elapsedMs": 325.8029170000009, + "nativeAssetMs": 28.39483300000211, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 874086400, + "cpuUserMs": 53.772, + "cpuSystemMs": 52.706, + "processPeakRssBytes": 875888640, + "rootPoolQueries": 5 + }, + { + "label": "warm-5", + "elapsedMs": 263.39350000000195, + "nativeAssetMs": 27.145415999999386, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 877297664, + "cpuUserMs": 46.555, + "cpuSystemMs": 58.031, + "processPeakRssBytes": 877297664, + "rootPoolQueries": 5 + }, + { + "label": "warm-6", + "elapsedMs": 258.43724999999904, + "nativeAssetMs": 24.80045799999789, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 877412352, + "cpuUserMs": 45.271, + "cpuSystemMs": 49.475, + "processPeakRssBytes": 877412352, + "rootPoolQueries": 5 + }, + { + "label": "warm-7", + "elapsedMs": 249.36533299999792, + "nativeAssetMs": 21.476291999999376, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 882917376, + "cpuUserMs": 40.674, + "cpuSystemMs": 57.454, + "processPeakRssBytes": 882917376, + "rootPoolQueries": 5 + }, + { + "label": "warm-8", + "elapsedMs": 238.76808299999902, + "nativeAssetMs": 19.84570800000074, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 885981184, + "cpuUserMs": 36.699, + "cpuSystemMs": 38.557, + "processPeakRssBytes": 885981184, + "rootPoolQueries": 5 + }, + { + "label": "warm-9", + "elapsedMs": 254.60462499999994, + "nativeAssetMs": 27.14166699999987, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886538240, + "cpuUserMs": 43.802, + "cpuSystemMs": 42.745, + "processPeakRssBytes": 886538240, + "rootPoolQueries": 5 + }, + { + "label": "warm-10", + "elapsedMs": 238.2076249999991, + "nativeAssetMs": 18.28241700000217, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886571008, + "cpuUserMs": 35.114, + "cpuSystemMs": 38.915, + "processPeakRssBytes": 886571008, + "rootPoolQueries": 5 + }, + { + "label": "warm-11", + "elapsedMs": 237.13320799999929, + "nativeAssetMs": 17.53529200000048, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886587392, + "cpuUserMs": 32.11, + "cpuSystemMs": 40.532, + "processPeakRssBytes": 886587392, + "rootPoolQueries": 5 + }, + { + "label": "warm-12", + "elapsedMs": 254.8944589999992, + "nativeAssetMs": 21.32683299999917, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886734848, + "cpuUserMs": 35.116, + "cpuSystemMs": 54.695, + "processPeakRssBytes": 886734848, + "rootPoolQueries": 5 + }, + { + "label": "warm-13", + "elapsedMs": 246.8540839999987, + "nativeAssetMs": 21.06829099999959, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886816768, + "cpuUserMs": 50.899, + "cpuSystemMs": 42.277, + "processPeakRssBytes": 886816768, + "rootPoolQueries": 5 + }, + { + "label": "warm-14", + "elapsedMs": 243.9827499999992, + "nativeAssetMs": 19.04475000000093, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 886833152, + "cpuUserMs": 35.215, + "cpuSystemMs": 43.516, + "processPeakRssBytes": 886833152, + "rootPoolQueries": 5 + }, + { + "label": "warm-15", + "elapsedMs": 238.93862499999886, + "nativeAssetMs": 18.254375000000437, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 861929472, + "cpuUserMs": 32.799, + "cpuSystemMs": 37.347, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-16", + "elapsedMs": 242.26750000000175, + "nativeAssetMs": 17.968958000001294, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 862502912, + "cpuUserMs": 37.265, + "cpuSystemMs": 36.262, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-17", + "elapsedMs": 231.48641700000007, + "nativeAssetMs": 15.712750000002416, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 864927744, + "cpuUserMs": 31.139, + "cpuSystemMs": 38.036, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-18", + "elapsedMs": 233.1889169999995, + "nativeAssetMs": 15.838167000001704, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 867467264, + "cpuUserMs": 27.639, + "cpuSystemMs": 38.248, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-19", + "elapsedMs": 238.24141699999745, + "nativeAssetMs": 14.957500000000437, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 867876864, + "cpuUserMs": 29.939, + "cpuSystemMs": 33.88, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-20", + "elapsedMs": 230.42920900000172, + "nativeAssetMs": 16.502959000001283, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 868384768, + "cpuUserMs": 28.371, + "cpuSystemMs": 38.99, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + }, + { + "label": "warm-new-service", + "elapsedMs": 232.0073749999974, + "nativeAssetMs": 15.748999999999796, + "sqlResponseBytes": 19288, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 869056512, + "cpuUserMs": 28.423, + "cpuSystemMs": 34.852, + "processPeakRssBytes": 886931456, + "rootPoolQueries": 5 + } + ], + "warmSummary": { + "count": 20, + "p50Ms": 245.41841699999895, + "p95Ms": 325.8029170000009, + "minMs": 230.42920900000172, + "maxMs": 403.48920799999905, + "medianCpuMs": 82.639 + } + }, + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "3f45986c6dd14f62b04e48b88d699b7c401d1ac877a894f7666696c84df8cc24", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 2, + "instanceDisk": { + "allocatedBytes": 152195072, + "fileBytes": 125969570, + "files": 6762 + }, + "repositoryHash": "fe0771eb3fea47e84689b61659ad5e4c320fb5ba77ca289bf8948220eb8acb83", + "repositoryFiles": 1350, + "runtimeFiles": 1351, + "runtimeBytes": 22377459, + "samples": [ + { + "label": "cold", + "elapsedMs": 6328.536042000001, + "nativeAssetMs": 1185.029708, + "sqlResponseBytes": 21018, + "objectGets": 2, + "objectReadBytes": 13450654, + "rssBytes": 864239616, + "cpuUserMs": 3393.371, + "cpuSystemMs": 3749.683, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 6 + }, + { + "label": "warm-1", + "elapsedMs": 296.3216250000005, + "nativeAssetMs": 45.2325409999994, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 897777664, + "cpuUserMs": 88.313, + "cpuSystemMs": 91.375, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-2", + "elapsedMs": 280.3572920000006, + "nativeAssetMs": 46.95887500000026, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 901021696, + "cpuUserMs": 78.114, + "cpuSystemMs": 95.396, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-3", + "elapsedMs": 276.8585409999996, + "nativeAssetMs": 42.61787499999991, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 901251072, + "cpuUserMs": 70.772, + "cpuSystemMs": 91.847, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-4", + "elapsedMs": 284.81624999999985, + "nativeAssetMs": 46.87012499999946, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 901365760, + "cpuUserMs": 67.437, + "cpuSystemMs": 97.816, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-5", + "elapsedMs": 283.0642910000006, + "nativeAssetMs": 44.04358300000058, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 901709824, + "cpuUserMs": 68.311, + "cpuSystemMs": 93.898, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-6", + "elapsedMs": 283.67491599999994, + "nativeAssetMs": 46.34595900000022, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 902168576, + "cpuUserMs": 69.452, + "cpuSystemMs": 91.08, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-7", + "elapsedMs": 302.01966699999684, + "nativeAssetMs": 47.38970799999879, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 902905856, + "cpuUserMs": 81.11, + "cpuSystemMs": 99.893, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-8", + "elapsedMs": 333.87829200000124, + "nativeAssetMs": 57.75833400000192, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 903921664, + "cpuUserMs": 82.534, + "cpuSystemMs": 110.13, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-9", + "elapsedMs": 332.312582999999, + "nativeAssetMs": 59.70783400000073, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 905183232, + "cpuUserMs": 86.169, + "cpuSystemMs": 133.898, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-10", + "elapsedMs": 309.3872909999991, + "nativeAssetMs": 57.14895799999795, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 905838592, + "cpuUserMs": 78.898, + "cpuSystemMs": 125.806, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-11", + "elapsedMs": 313.51016700000037, + "nativeAssetMs": 62.051665999999386, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 906330112, + "cpuUserMs": 81.977, + "cpuSystemMs": 130.557, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-12", + "elapsedMs": 326.54258400000253, + "nativeAssetMs": 74.06966600000305, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 906952704, + "cpuUserMs": 72.584, + "cpuSystemMs": 102.054, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-13", + "elapsedMs": 294.62420900000143, + "nativeAssetMs": 50.54508299999725, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 907395072, + "cpuUserMs": 71.091, + "cpuSystemMs": 104.148, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-14", + "elapsedMs": 291.966292000001, + "nativeAssetMs": 49.89749999999913, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 907902976, + "cpuUserMs": 75.121, + "cpuSystemMs": 98.96, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-15", + "elapsedMs": 278.43558400000256, + "nativeAssetMs": 43.16858299999876, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 908443648, + "cpuUserMs": 70.947, + "cpuSystemMs": 91.63, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-16", + "elapsedMs": 280.7341250000027, + "nativeAssetMs": 43.70062499999767, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 908951552, + "cpuUserMs": 65.543, + "cpuSystemMs": 91.628, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-17", + "elapsedMs": 307.03354200000103, + "nativeAssetMs": 58.58600000000297, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 909524992, + "cpuUserMs": 75.961, + "cpuSystemMs": 114.523, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-18", + "elapsedMs": 304.9934580000008, + "nativeAssetMs": 53.50358300000153, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 909901824, + "cpuUserMs": 70.672, + "cpuSystemMs": 105.85, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-19", + "elapsedMs": 298.8248749999984, + "nativeAssetMs": 52.481667000000016, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 910409728, + "cpuUserMs": 69.146, + "cpuSystemMs": 112.195, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-20", + "elapsedMs": 325.0633339999986, + "nativeAssetMs": 52.81354099999953, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 910901248, + "cpuUserMs": 65.785, + "cpuSystemMs": 106.417, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + }, + { + "label": "warm-new-service", + "elapsedMs": 301.2625000000007, + "nativeAssetMs": 53.47154199999932, + "sqlResponseBytes": 20312, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 911409152, + "cpuUserMs": 71.72, + "cpuSystemMs": 112.835, + "processPeakRssBytes": 936968192, + "rootPoolQueries": 5 + } + ], + "warmSummary": { + "count": 20, + "p50Ms": 297.57324999999946, + "p95Ms": 332.312582999999, + "minMs": 276.8585409999996, + "maxMs": 333.87829200000124, + "medianCpuMs": 174.93849999999998 + } + }, + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "3f45986c6dd14f62b04e48b88d699b7c401d1ac877a894f7666696c84df8cc24", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 10, + "instanceDisk": { + "allocatedBytes": 596471808, + "fileBytes": 485644240, + "files": 28562 + }, + "repositoryHash": "8905e1b5adbdbc1f4be27131dcc3065dddbe615f73e90f8e0ecd9a6349feab43", + "repositoryFiles": 1358, + "runtimeFiles": 1359, + "runtimeBytes": 22378235, + "samples": [ + { + "label": "cold", + "elapsedMs": 11038.252875000002, + "nativeAssetMs": 2835.388499999999, + "sqlResponseBytes": 29242, + "objectGets": 2, + "objectReadBytes": 13452324, + "rssBytes": 1175388160, + "cpuUserMs": 10688.652, + "cpuSystemMs": 12125.498, + "processPeakRssBytes": 1175388160, + "rootPoolQueries": 6 + }, + { + "label": "warm-1", + "elapsedMs": 509.059874999999, + "nativeAssetMs": 206.29983300000094, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1200144384, + "cpuUserMs": 274.241, + "cpuSystemMs": 450.411, + "processPeakRssBytes": 1200144384, + "rootPoolQueries": 5 + }, + { + "label": "warm-2", + "elapsedMs": 563.2716670000009, + "nativeAssetMs": 229.46791700000176, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1205272576, + "cpuUserMs": 306.689, + "cpuSystemMs": 498.681, + "processPeakRssBytes": 1205272576, + "rootPoolQueries": 5 + }, + { + "label": "warm-3", + "elapsedMs": 551.9728749999995, + "nativeAssetMs": 224.04054199999882, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1208942592, + "cpuUserMs": 287.219, + "cpuSystemMs": 511.782, + "processPeakRssBytes": 1208942592, + "rootPoolQueries": 5 + }, + { + "label": "warm-4", + "elapsedMs": 540.7157499999994, + "nativeAssetMs": 213.42754199999763, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1212006400, + "cpuUserMs": 276.609, + "cpuSystemMs": 480.152, + "processPeakRssBytes": 1212006400, + "rootPoolQueries": 5 + }, + { + "label": "warm-5", + "elapsedMs": 578.5833329999987, + "nativeAssetMs": 238.3347500000018, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216577536, + "cpuUserMs": 304.291, + "cpuSystemMs": 483.345, + "processPeakRssBytes": 1216577536, + "rootPoolQueries": 5 + }, + { + "label": "warm-6", + "elapsedMs": 524.1193330000024, + "nativeAssetMs": 206.63066600000093, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1220493312, + "cpuUserMs": 262.339, + "cpuSystemMs": 476.115, + "processPeakRssBytes": 1220493312, + "rootPoolQueries": 5 + }, + { + "label": "warm-7", + "elapsedMs": 522.1358340000006, + "nativeAssetMs": 203.664249999998, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1223901184, + "cpuUserMs": 255.151, + "cpuSystemMs": 504.054, + "processPeakRssBytes": 1223901184, + "rootPoolQueries": 5 + }, + { + "label": "warm-8", + "elapsedMs": 536.1341250000005, + "nativeAssetMs": 223.24095900000248, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1228275712, + "cpuUserMs": 269.799, + "cpuSystemMs": 495.782, + "processPeakRssBytes": 1228275712, + "rootPoolQueries": 5 + }, + { + "label": "warm-9", + "elapsedMs": 527.7411670000001, + "nativeAssetMs": 207.70304100000067, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1232420864, + "cpuUserMs": 275.873, + "cpuSystemMs": 475.242, + "processPeakRssBytes": 1232420864, + "rootPoolQueries": 5 + }, + { + "label": "warm-10", + "elapsedMs": 535.4337919999998, + "nativeAssetMs": 221.5627080000013, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1227620352, + "cpuUserMs": 277.786, + "cpuSystemMs": 552.621, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-11", + "elapsedMs": 585.1106249999975, + "nativeAssetMs": 240.04220799999894, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1228800000, + "cpuUserMs": 311.691, + "cpuSystemMs": 525.373, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-12", + "elapsedMs": 582.450458000003, + "nativeAssetMs": 240.56300000000192, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1228947456, + "cpuUserMs": 319.294, + "cpuSystemMs": 517.277, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-13", + "elapsedMs": 1249.3626670000012, + "nativeAssetMs": 432.052208000001, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1229373440, + "cpuUserMs": 359.244, + "cpuSystemMs": 551.464, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-14", + "elapsedMs": 599.4650000000001, + "nativeAssetMs": 259.7930839999972, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1229897728, + "cpuUserMs": 321.251, + "cpuSystemMs": 539.465, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-15", + "elapsedMs": 674.1641249999993, + "nativeAssetMs": 298.9093750000029, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1229963264, + "cpuUserMs": 368.737, + "cpuSystemMs": 617.768, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-16", + "elapsedMs": 636.989166000003, + "nativeAssetMs": 294.70037500000035, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1230864384, + "cpuUserMs": 345.605, + "cpuSystemMs": 578.557, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-17", + "elapsedMs": 613.3584579999988, + "nativeAssetMs": 264.15899999999965, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1231388672, + "cpuUserMs": 336.042, + "cpuSystemMs": 511.512, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-18", + "elapsedMs": 592.2004589999997, + "nativeAssetMs": 239.67762499999662, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216462848, + "cpuUserMs": 427.338, + "cpuSystemMs": 523.325, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-19", + "elapsedMs": 584.3585830000011, + "nativeAssetMs": 250.61229200000525, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216462848, + "cpuUserMs": 316.455, + "cpuSystemMs": 520.297, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-20", + "elapsedMs": 566.4432920000036, + "nativeAssetMs": 226.26541700000234, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216462848, + "cpuUserMs": 297.778, + "cpuSystemMs": 511.475, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + }, + { + "label": "warm-new-service", + "elapsedMs": 552.4209159999955, + "nativeAssetMs": 222.31191699999908, + "sqlResponseBytes": 28536, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1216643072, + "cpuUserMs": 292.34, + "cpuSystemMs": 521.954, + "processPeakRssBytes": 1235435520, + "rootPoolQueries": 5 + } + ], + "warmSummary": { + "count": 20, + "p50Ms": 572.5133125000011, + "p95Ms": 674.1641249999993, + "minMs": 509.059874999999, + "maxMs": 1249.3626670000012, + "medianCpuMs": 819.8299999999999 + } + } + ], + "s3Proof": { + "server": "MinIO RELEASE.2025-04-22T22-12-26Z built from official source", + "files": 22, + "archiveBytes": 18008626, + "decodedBytes": 18000095, + "multipartRoundTrip": true, + "coldRestore": true, + "verifiedDurable": true + }, + "automaticLifecycleVerification": { + "date": "2026-10-28", + "notes": [ + "Final implementation with automatic conversion, exact-commit acquisition, native asset expiry and repair.", + "PAPERCLIP_TEST_POSTGRES_TEMPLATE=0 bypassed a shared test-template startup failure; fresh disposable PostgreSQL clusters were used.", + "Same synthetic input, 5 ms emulated network delay each way, authorization, unrelated-library shape, and native asset delivery as the earlier measurements.", + "Host was shared with other work. Timings are local regression samples, not production latency guarantees." + ], + "summaries": [ + { + "selectedSkills": 1, + "warmSamples": 20, + "medianMs": 256.1131045000002, + "p95Ms": 302.01637499999924, + "rootPoolQueries": [ + 5 + ], + "sqlResponseBytes": [ + 19423 + ], + "objectGets": 0, + "objectReadBytes": 0, + "coldMs": 5521.755166999999, + "sourceChangesHash": "46a4a05b61b399d18d1009085b6e54e9853563af98b3580318fb69290f0e05f1" + }, + { + "selectedSkills": 10, + "warmSamples": 20, + "medianMs": 690.6289374999978, + "p95Ms": 815.8250839999964, + "rootPoolQueries": [ + 5 + ], + "sqlResponseBytes": [ + 28969 + ], + "objectGets": 0, + "objectReadBytes": 0, + "coldMs": 47645.3685, + "sourceChangesHash": "46a4a05b61b399d18d1009085b6e54e9853563af98b3580318fb69290f0e05f1" + } + ], + "measurements": [ + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "46a4a05b61b399d18d1009085b6e54e9853563af98b3580318fb69290f0e05f1", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 1, + "instanceDisk": { + "allocatedBytes": 96739328, + "fileBytes": 81013397, + "files": 4056 + }, + "repositoryHash": "5cedce49bfb80e1430df76f4b3889a1b20095545e9aeb575d1b037ab3c6ecdbe", + "repositoryFiles": 1349, + "runtimeFiles": 1350, + "runtimeBytes": 22377362, + "samples": [ + { + "label": "cold", + "elapsedMs": 5521.755166999999, + "nativeAssetMs": 2862.993250000003, + "sqlResponseBytes": 20129, + "objectGets": 2, + "objectReadBytes": 13450429, + "rssBytes": 839041024, + "cpuUserMs": 2907.478, + "cpuSystemMs": 2180.612, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 6, + "traceId": "d930e609f6524d003325bbab3cd24fff", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 123.332, + "spanId": "575fff8ba56923bc", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + }, + { + "name": "skills.rows", + "durationMs": 24.519875, + "spanId": "691e337e466035ff", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + }, + { + "name": "skills.version.read", + "durationMs": 24.449625, + "spanId": "52d8b8322c1f9fbf", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.714625, + "spanId": "7b3663ddbf76629f", + "parentSpanId": "2271436e8b6ebccf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 2415.632125, + "spanId": "2271436e8b6ebccf", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + }, + { + "name": "skills.consume", + "durationMs": 2884.642125, + "spanId": "91f51464e883bbeb", + "parentSpanId": "b50455a614608bbf", + "attributes": { + "paperclip.task.run.run_id": "2cc2920de94b" + } + } + ] + }, + { + "label": "warm-1", + "elapsedMs": 277.8793340000011, + "nativeAssetMs": 25.730290999999852, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 859815936, + "cpuUserMs": 68.572, + "cpuSystemMs": 48.415, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "17f51b0a84f58bf1add1d37a9529d0d9", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 105.477125, + "spanId": "e61340dcd3f750f1", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + }, + { + "name": "skills.rows", + "durationMs": 22.589, + "spanId": "bedf6f5708ca6865", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + }, + { + "name": "skills.version.read", + "durationMs": 24.176708, + "spanId": "b01f550b571c79df", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.274042, + "spanId": "1f6045577c1ce282", + "parentSpanId": "0f478bd06304aa65", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 38.816041, + "spanId": "0f478bd06304aa65", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + }, + { + "name": "skills.consume", + "durationMs": 50.267792, + "spanId": "3f191c4b1c5f2655", + "parentSpanId": "29463685a953427a", + "attributes": { + "paperclip.task.run.run_id": "d458c6d64f4f" + } + } + ] + }, + { + "label": "warm-2", + "elapsedMs": 258.88416700000016, + "nativeAssetMs": 22.96679100000256, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860487680, + "cpuUserMs": 60.559, + "cpuSystemMs": 47.455, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "baa58a8abe13ada2e7b5f888f1b543ce", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.164125, + "spanId": "039f5ed5d1f5b830", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + }, + { + "name": "skills.rows", + "durationMs": 23.490791, + "spanId": "e621641b17743498", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + }, + { + "name": "skills.version.read", + "durationMs": 21.576416, + "spanId": "4db591c9e0feecb3", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.304541, + "spanId": "6878c2b616698075", + "parentSpanId": "35aeb337e724e325", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 38.412708, + "spanId": "35aeb337e724e325", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + }, + { + "name": "skills.consume", + "durationMs": 46.015041, + "spanId": "bbb3f00f9b4bfdf3", + "parentSpanId": "a8bf64c1496704a2", + "attributes": { + "paperclip.task.run.run_id": "b408728c2c11" + } + } + ] + }, + { + "label": "warm-3", + "elapsedMs": 256.2706670000007, + "nativeAssetMs": 20.635957999998936, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860504064, + "cpuUserMs": 52.112, + "cpuSystemMs": 39.375, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "e5ff1facc7bda4e65d75f4e7549bff8c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 92.212625, + "spanId": "dca51402638b670c", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + }, + { + "name": "skills.rows", + "durationMs": 22.964166, + "spanId": "0439f29191eac439", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.306042, + "spanId": "466f10562fc4caf7", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.067, + "spanId": "9f9e84fd9ef963eb", + "parentSpanId": "d5fbca670b5f91dd", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 37.148208, + "spanId": "d5fbca670b5f91dd", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + }, + { + "name": "skills.consume", + "durationMs": 43.428084, + "spanId": "9f55673f9ed3273f", + "parentSpanId": "177fc240f2c59960", + "attributes": { + "paperclip.task.run.run_id": "e6864ac5a9a0" + } + } + ] + }, + { + "label": "warm-4", + "elapsedMs": 252.9054580000011, + "nativeAssetMs": 21.87449999999808, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860553216, + "cpuUserMs": 48.459, + "cpuSystemMs": 39.237, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "ef8a05c329b3e2336fc3486592050728", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 89.006375, + "spanId": "e85a4aadbd4feccc", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + }, + { + "name": "skills.rows", + "durationMs": 22.330958, + "spanId": "f850ca66e27a29b1", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.321, + "spanId": "0d26d736bd91d71d", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.522458, + "spanId": "1d8bbd402e3d0c01", + "parentSpanId": "6ceccd78ea3bc47e", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 37.512292, + "spanId": "6ceccd78ea3bc47e", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + }, + { + "name": "skills.consume", + "durationMs": 44.858833, + "spanId": "37c073e9f09fed4d", + "parentSpanId": "d414ae5405792578", + "attributes": { + "paperclip.task.run.run_id": "a247c941ad7f" + } + } + ] + }, + { + "label": "warm-5", + "elapsedMs": 244.24670799999876, + "nativeAssetMs": 19.243334000002505, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860618752, + "cpuUserMs": 38.593, + "cpuSystemMs": 34.984, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "2bd5af6006769dd2721612d19cf0e7e5", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 86.527541, + "spanId": "27cb1035d28b6263", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + }, + { + "name": "skills.rows", + "durationMs": 22.759084, + "spanId": "e542b8f3f97b7625", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.41175, + "spanId": "f02d61a8f91295fd", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.862125, + "spanId": "9435bcb01b542105", + "parentSpanId": "f973c621f62205ec", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 35.202375, + "spanId": "f973c621f62205ec", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + }, + { + "name": "skills.consume", + "durationMs": 41.587958, + "spanId": "0af0696783146f45", + "parentSpanId": "de008da6630edb43", + "attributes": { + "paperclip.task.run.run_id": "490e02151c3f" + } + } + ] + }, + { + "label": "warm-6", + "elapsedMs": 246.31941700000243, + "nativeAssetMs": 20.167625000001863, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860651520, + "cpuUserMs": 40.133, + "cpuSystemMs": 35.96, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "e9d381b984412682ee0d054785f5bc07", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 88.371584, + "spanId": "d7f9a9d6d1f6a9f7", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + }, + { + "name": "skills.rows", + "durationMs": 23.367584, + "spanId": "83b30eb0d37f3d43", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + }, + { + "name": "skills.version.read", + "durationMs": 21.615917, + "spanId": "20f08029124c778a", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.984708, + "spanId": "2ff9b56bf7d54e5d", + "parentSpanId": "6f5238c569424956", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 35.582, + "spanId": "6f5238c569424956", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + }, + { + "name": "skills.consume", + "durationMs": 43.176875, + "spanId": "0f1bcf11f3cb5b88", + "parentSpanId": "3f92ac88fa693c72", + "attributes": { + "paperclip.task.run.run_id": "817cd6cebdb7" + } + } + ] + }, + { + "label": "warm-7", + "elapsedMs": 260.0483329999988, + "nativeAssetMs": 20.238124999999854, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860651520, + "cpuUserMs": 40.479, + "cpuSystemMs": 42.819, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "0e7b563ae50b82c18c950d14822a0238", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 87.678792, + "spanId": "1a0f8d270b4ae1c2", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + }, + { + "name": "skills.rows", + "durationMs": 23.014667, + "spanId": "61e2f4ed73027123", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.224416, + "spanId": "a27fd4a5b6d6f50d", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + }, + { + "name": "skills.version.verify", + "durationMs": 23.332666, + "spanId": "20a9c335269146ad", + "parentSpanId": "6ae03d2c64c54861", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 50.015208, + "spanId": "6ae03d2c64c54861", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + }, + { + "name": "skills.consume", + "durationMs": 41.842834, + "spanId": "ac49533cdf6e0aba", + "parentSpanId": "9b0d42c4199999fe", + "attributes": { + "paperclip.task.run.run_id": "8f00f3f7e626" + } + } + ] + }, + { + "label": "warm-8", + "elapsedMs": 296.3832079999993, + "nativeAssetMs": 23.272417000000132, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 860717056, + "cpuUserMs": 44.182, + "cpuSystemMs": 40.068, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "b313a786abc8784f76eebb04f2b65fd1", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.983125, + "spanId": "01327dfcea49dd05", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + }, + { + "name": "skills.rows", + "durationMs": 23.028041, + "spanId": "3419633149135de9", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.623292, + "spanId": "7a5993593c7b5a2a", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 39.360792, + "spanId": "86ff67922c241ade", + "parentSpanId": "0389067fc98ed3fd", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 76.266167, + "spanId": "0389067fc98ed3fd", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + }, + { + "name": "skills.consume", + "durationMs": 46.797167, + "spanId": "b6de6fa36685f548", + "parentSpanId": "969c7bb79a095fed", + "attributes": { + "paperclip.task.run.run_id": "5ad83b0a4c7e" + } + } + ] + }, + { + "label": "warm-9", + "elapsedMs": 249.17170800000167, + "nativeAssetMs": 21.109082999999373, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 861306880, + "cpuUserMs": 44.78, + "cpuSystemMs": 38.28, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "8b174d18f9de1da799551b952d21375c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.344708, + "spanId": "050896cedad88644", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + }, + { + "name": "skills.rows", + "durationMs": 23.4595, + "spanId": "099e4789f58db9be", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.737334, + "spanId": "899d373a466f0ee4", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.478084, + "spanId": "c003f06a34d9fd6b", + "parentSpanId": "656625d058c3f76c", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 33.079208, + "spanId": "656625d058c3f76c", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + }, + { + "name": "skills.consume", + "durationMs": 44.442958, + "spanId": "16ada23b6310af02", + "parentSpanId": "a7d438170be08a3e", + "attributes": { + "paperclip.task.run.run_id": "2edf2adb7a31" + } + } + ] + }, + { + "label": "warm-10", + "elapsedMs": 246.82358399999794, + "nativeAssetMs": 18.57062500000029, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 832356352, + "cpuUserMs": 37.233, + "cpuSystemMs": 34.641, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "f8bda941e19a7e3e55eb6ba599c8dc64", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.506625, + "spanId": "a89bd12424ddcc52", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + }, + { + "name": "skills.rows", + "durationMs": 24.072709, + "spanId": "1c7e14d1d64a7c1b", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.888542, + "spanId": "1aae1cba9874b9a8", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.823833, + "spanId": "e6dbd7401dd06462", + "parentSpanId": "6cf81c1ffc2675ec", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 33.593625, + "spanId": "6cf81c1ffc2675ec", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + }, + { + "name": "skills.consume", + "durationMs": 41.207167, + "spanId": "5e54082e0a3da96d", + "parentSpanId": "6de7234baba7d0a7", + "attributes": { + "paperclip.task.run.run_id": "3301adc9e285" + } + } + ] + }, + { + "label": "warm-11", + "elapsedMs": 247.22716699999728, + "nativeAssetMs": 18.73529200000121, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 832847872, + "cpuUserMs": 34.961, + "cpuSystemMs": 35.58, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "bc43ae232b084940d1d2916542b90f38", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.696958, + "spanId": "f761332966b03494", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + }, + { + "name": "skills.rows", + "durationMs": 23.352083, + "spanId": "96bf5714ed810b6c", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.087541, + "spanId": "e1f97ad53c5fa76f", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.558459, + "spanId": "6f379bc6aebb7259", + "parentSpanId": "5d12ce05f1977b62", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 33.8355, + "spanId": "5d12ce05f1977b62", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + }, + { + "name": "skills.consume", + "durationMs": 41.15275, + "spanId": "aba50da7f24b77f6", + "parentSpanId": "596be1d256dd1361", + "attributes": { + "paperclip.task.run.run_id": "bcc55a8e9692" + } + } + ] + }, + { + "label": "warm-12", + "elapsedMs": 246.921957999999, + "nativeAssetMs": 20.732458999998926, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 834371584, + "cpuUserMs": 37.99, + "cpuSystemMs": 39.364, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "104eae77376a1d6eed770793f6005b80", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.51825, + "spanId": "0d615cb98e361910", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + }, + { + "name": "skills.rows", + "durationMs": 23.632041, + "spanId": "cb25e039dd457704", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + }, + { + "name": "skills.version.read", + "durationMs": 21.565334, + "spanId": "94853b0c375376ff", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.611708, + "spanId": "b9c7020271c407dc", + "parentSpanId": "f4e9f25aad1cbc1c", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 32.759458, + "spanId": "f4e9f25aad1cbc1c", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + }, + { + "name": "skills.consume", + "durationMs": 44.410834, + "spanId": "8746f9a0a7a83f1b", + "parentSpanId": "9c315ea8073b3cba", + "attributes": { + "paperclip.task.run.run_id": "1a1942f39467" + } + } + ] + }, + { + "label": "warm-13", + "elapsedMs": 248.8155000000006, + "nativeAssetMs": 20.55420899999808, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 835338240, + "cpuUserMs": 49.511, + "cpuSystemMs": 40.047, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "a4e34ea74e542a06498eb2c0bfdb7d78", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 87.402583, + "spanId": "6bd7e492a84a3d62", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + }, + { + "name": "skills.rows", + "durationMs": 23.381709, + "spanId": "69786f2248b675e9", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + }, + { + "name": "skills.version.read", + "durationMs": 21.377708, + "spanId": "e761580f607cff81", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.884625, + "spanId": "cebe63bb7ed1775f", + "parentSpanId": "532261ccffb31019", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.704917, + "spanId": "532261ccffb31019", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + }, + { + "name": "skills.consume", + "durationMs": 43.189333, + "spanId": "6fcefc7fab230e8c", + "parentSpanId": "41d1cb77faa41d10", + "attributes": { + "paperclip.task.run.run_id": "fb770e7c9aa8" + } + } + ] + }, + { + "label": "warm-14", + "elapsedMs": 254.31112500000017, + "nativeAssetMs": 21.776166999999987, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 837812224, + "cpuUserMs": 44.22, + "cpuSystemMs": 37.706, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "09058904f6838d35fab91829f2cede1c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 91.188416, + "spanId": "48bb96823338e726", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + }, + { + "name": "skills.rows", + "durationMs": 21.949208, + "spanId": "d42c2544cc22b1b1", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.344958, + "spanId": "1d2d49b58286e326", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.674125, + "spanId": "8ae309fa7a8ae97c", + "parentSpanId": "bc3c6b564be907eb", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.717125, + "spanId": "bc3c6b564be907eb", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + }, + { + "name": "skills.consume", + "durationMs": 45.310375, + "spanId": "9eb0e8a23aaa004c", + "parentSpanId": "fd03100985023afe", + "attributes": { + "paperclip.task.run.run_id": "addf40db044a" + } + } + ] + }, + { + "label": "warm-15", + "elapsedMs": 256.34162500000093, + "nativeAssetMs": 21.958875000000262, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 838369280, + "cpuUserMs": 40.477, + "cpuSystemMs": 40.68, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "85e9e33244624553ab1ef014fcbc202b", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 91.318042, + "spanId": "bb22f11d4bbe41e8", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + }, + { + "name": "skills.rows", + "durationMs": 24.197083, + "spanId": "c799b894086f66c6", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.747458, + "spanId": "9cb5bf472976a8e2", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.437792, + "spanId": "53faecf9fc592d09", + "parentSpanId": "241fd52adb2211bf", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.593166, + "spanId": "241fd52adb2211bf", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + }, + { + "name": "skills.consume", + "durationMs": 44.317041, + "spanId": "41fb8559f4d29765", + "parentSpanId": "2204226173a9329b", + "attributes": { + "paperclip.task.run.run_id": "38b0c7fd276c" + } + } + ] + }, + { + "label": "warm-16", + "elapsedMs": 361.1522499999992, + "nativeAssetMs": 56.873374999999214, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 838828032, + "cpuUserMs": 45.728, + "cpuSystemMs": 34.21, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "5f24f8506e0b6fddedde523af9afa4f5", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 88.929625, + "spanId": "7a196c77e19eff83", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + }, + { + "name": "skills.rows", + "durationMs": 22.999291, + "spanId": "cd320c350c1f601a", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.893541, + "spanId": "34dccbf348b6dd3f", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + }, + { + "name": "skills.version.verify", + "durationMs": 56.2555, + "spanId": "0423cf859794a3cf", + "parentSpanId": "bc4ec6d07d341dfd", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 105.091542, + "spanId": "bc4ec6d07d341dfd", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + }, + { + "name": "skills.consume", + "durationMs": 84.441625, + "spanId": "a5a7081c6c2961ec", + "parentSpanId": "18ef2bceda700d76", + "attributes": { + "paperclip.task.run.run_id": "cc2135b4058a" + } + } + ] + }, + { + "label": "warm-17", + "elapsedMs": 257.58879100000195, + "nativeAssetMs": 24.06116699999984, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 839352320, + "cpuUserMs": 42.417, + "cpuSystemMs": 43.725, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "1a7b8b693b4c4e85be0773b6a3d522a5", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 91.829958, + "spanId": "0966be6de1c97482", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + }, + { + "name": "skills.rows", + "durationMs": 22.791625, + "spanId": "f1a2aa94c5da75b7", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + }, + { + "name": "skills.version.read", + "durationMs": 23.146834, + "spanId": "620f70dc51819b0d", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.523083, + "spanId": "8a041b6acb40ffd2", + "parentSpanId": "0f2acbbee58bcdef", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 37.086292, + "spanId": "0f2acbbee58bcdef", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + }, + { + "name": "skills.consume", + "durationMs": 46.412834, + "spanId": "b7d82ae57fc8b0da", + "parentSpanId": "06b9fc3ee6bb8718", + "attributes": { + "paperclip.task.run.run_id": "0909450fadd7" + } + } + ] + }, + { + "label": "warm-18", + "elapsedMs": 261.34670899999765, + "nativeAssetMs": 24.197124999998778, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 839811072, + "cpuUserMs": 41.926, + "cpuSystemMs": 47.502, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "161d8b331b6a5eb16c4703c7df25aa22", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.52425, + "spanId": "450dda07090d21e0", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + }, + { + "name": "skills.rows", + "durationMs": 24.316542, + "spanId": "cd3ff72582764824", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.990042, + "spanId": "ecb7218f0217a931", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.900792, + "spanId": "302f69462c259eb5", + "parentSpanId": "76d6fb44c9bd9169", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.195083, + "spanId": "76d6fb44c9bd9169", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + }, + { + "name": "skills.consume", + "durationMs": 48.113084, + "spanId": "fa7076037887129b", + "parentSpanId": "db6c6cae4bde6b5f", + "attributes": { + "paperclip.task.run.run_id": "c7b4f70c2f82" + } + } + ] + }, + { + "label": "warm-19", + "elapsedMs": 255.9555419999997, + "nativeAssetMs": 22.198041999999987, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 840302592, + "cpuUserMs": 41.007, + "cpuSystemMs": 42.203, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "d6aa9695bfab2522a5b20285da1fb598", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 92.449917, + "spanId": "d46a9727b1ba1a04", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + }, + { + "name": "skills.rows", + "durationMs": 21.752667, + "spanId": "439112b181436b91", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + }, + { + "name": "skills.version.read", + "durationMs": 22.827208, + "spanId": "3aae300a50f483ec", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.7435, + "spanId": "30ccb2e21a60587b", + "parentSpanId": "5b0d15bafaa93022", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.64025, + "spanId": "5b0d15bafaa93022", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + }, + { + "name": "skills.consume", + "durationMs": 45.945459, + "spanId": "4f855ccc5cd02e6e", + "parentSpanId": "800bde181862ad1a", + "attributes": { + "paperclip.task.run.run_id": "b938e1041e77" + } + } + ] + }, + { + "label": "warm-20", + "elapsedMs": 302.01637499999924, + "nativeAssetMs": 55.15375000000131, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 840957952, + "cpuUserMs": 44.687, + "cpuSystemMs": 37.404, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "a65908be572847cf5d56c15739917e1e", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.346583, + "spanId": "b7578db1eec6e06b", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + }, + { + "name": "skills.rows", + "durationMs": 22.568333, + "spanId": "5cea64ceb62c4e03", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + }, + { + "name": "skills.version.read", + "durationMs": 24.110416, + "spanId": "f70e0dbbad2cd917", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + }, + { + "name": "skills.version.verify", + "durationMs": 21.076459, + "spanId": "7edc1debda754122", + "parentSpanId": "9834693c11018354", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 46.041083, + "spanId": "9834693c11018354", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + }, + { + "name": "skills.consume", + "durationMs": 79.26225, + "spanId": "0692bf561c7e0419", + "parentSpanId": "6b9b2ae703f098b3", + "attributes": { + "paperclip.task.run.run_id": "c35b83889670" + } + } + ] + }, + { + "label": "warm-new-service", + "elapsedMs": 405.37420900000143, + "nativeAssetMs": 59.024374999997235, + "sqlResponseBytes": 19423, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 841891840, + "cpuUserMs": 59.181, + "cpuSystemMs": 60.108, + "processPeakRssBytes": 871350272, + "rootPoolQueries": 5, + "traceId": "1d872a742343629c5e2a9550fce4e574", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 119.922875, + "spanId": "24ec4140ecbe6008", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + }, + { + "name": "skills.rows", + "durationMs": 31.560959, + "spanId": "f2247ca794d615c5", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + }, + { + "name": "skills.version.read", + "durationMs": 25.014542, + "spanId": "96aa7ea84f7a016a", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + }, + { + "name": "skills.version.verify", + "durationMs": 71.03075, + "spanId": "d10a2a22d516cfc4", + "parentSpanId": "4b5e1ed80d6d5e2b", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65", + "paperclip.execution.fileCount": 1350 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 108.683375, + "spanId": "4b5e1ed80d6d5e2b", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + }, + { + "name": "skills.consume", + "durationMs": 84.205833, + "spanId": "5da070654b83f17a", + "parentSpanId": "56b6968f5bd38c1c", + "attributes": { + "paperclip.task.run.run_id": "6e7592a7bd65" + } + } + ] + } + ] + }, + { + "schema": "paperclip.native-skill-preparation-benchmark.v1", + "sourceRevision": "1ed0af41867ba40fd1d23292d395588d18c2fcbe", + "sourceChangesHash": "46a4a05b61b399d18d1009085b6e54e9853563af98b3580318fb69290f0e05f1", + "exportedOtlp": false, + "source": "supplied immutable snapshot", + "networkDelayMsPerDirection": 5, + "authenticated": true, + "stagingShape": true, + "externalPackages": true, + "nativeAssets": true, + "selectedSkillCount": 10, + "instanceDisk": { + "allocatedBytes": 596471808, + "fileBytes": 485644240, + "files": 28572 + }, + "repositoryHash": "8905e1b5adbdbc1f4be27131dcc3065dddbe615f73e90f8e0ecd9a6349feab43", + "repositoryFiles": 1358, + "runtimeFiles": 1359, + "runtimeBytes": 22378235, + "samples": [ + { + "label": "cold", + "elapsedMs": 47645.3685, + "nativeAssetMs": 9308.106708, + "sqlResponseBytes": 29675, + "objectGets": 2, + "objectReadBytes": 13452324, + "rssBytes": 1166737408, + "cpuUserMs": 14069.023, + "cpuSystemMs": 17674.382, + "processPeakRssBytes": 1167867904, + "rootPoolQueries": 6, + "traceId": "562419a79efb8a11c006e311ca5593ad", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 114.230334, + "spanId": "8704b7082bbc332a", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.rows", + "durationMs": 28.7575, + "spanId": "8230d5d009e02885", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.125667, + "spanId": "50abf4ae64969ac3", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 2.812667, + "spanId": "5ccd1e0662caa241", + "parentSpanId": "076b8e8a5a95e497", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 3290.498, + "spanId": "076b8e8a5a95e497", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.060542, + "spanId": "e4c3915d46470f88", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.343125, + "spanId": "a246125338ced67b", + "parentSpanId": "3b97ef532785fd9b", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 2141.325542, + "spanId": "3b97ef532785fd9b", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.035167, + "spanId": "a2d82825e93112fe", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.268208, + "spanId": "0dbc0869fe1e7836", + "parentSpanId": "076ec67ef4c0e4b0", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 2875.915042, + "spanId": "076ec67ef4c0e4b0", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025917, + "spanId": "24db2198f4633e63", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.435417, + "spanId": "ce6761e421a0df42", + "parentSpanId": "f39908478d906933", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 3134.053334, + "spanId": "f39908478d906933", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024083, + "spanId": "907c1d86af17117d", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.440625, + "spanId": "a0ee9581c0f3865e", + "parentSpanId": "0de953dd6f306e20", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 3681.356417, + "spanId": "0de953dd6f306e20", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.038791, + "spanId": "93dd31171c070a8d", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.868, + "spanId": "c6614f7e35241394", + "parentSpanId": "421dc58c47a3c8b5", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 3857.3914999999997, + "spanId": "421dc58c47a3c8b5", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033833, + "spanId": "de20a361e93cc987", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 5.258875, + "spanId": "ea67abd8cae97096", + "parentSpanId": "f0afadbef89a51e1", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 8359.825959, + "spanId": "f0afadbef89a51e1", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033667, + "spanId": "4f3e15656690747c", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.254625, + "spanId": "e65c14f6a406806a", + "parentSpanId": "69f6cbb9eca241f5", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 5134.679292, + "spanId": "69f6cbb9eca241f5", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.045833, + "spanId": "b3c1b5f43355b889", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.758916, + "spanId": "c4f7a7232789b46e", + "parentSpanId": "b560806bec319f1c", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 4453.032834, + "spanId": "b560806bec319f1c", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019167, + "spanId": "0a7209360c8ac971", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.version.verify", + "durationMs": 0.1945, + "spanId": "d97a270982776d3d", + "parentSpanId": "9a10a4cff81546bb", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 1135.038417, + "spanId": "9a10a4cff81546bb", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + }, + { + "name": "skills.consume", + "durationMs": 9338.957667, + "spanId": "e8119d7dca8e3b15", + "parentSpanId": "07885bfad81d0673", + "attributes": { + "paperclip.task.run.run_id": "e1cb14df030f" + } + } + ] + }, + { + "label": "warm-1", + "elapsedMs": 745.369000000006, + "nativeAssetMs": 284.55312499999127, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170554880, + "cpuUserMs": 380.795, + "cpuSystemMs": 509.73, + "processPeakRssBytes": 1170554880, + "rootPoolQueries": 5, + "traceId": "75dd1e1c08d25c152d7d912a045c0fbe", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 107.59175, + "spanId": "8d3a42ad2215ac17", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.rows", + "durationMs": 51.915208, + "spanId": "c2817ffff0eb306f", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.039083, + "spanId": "10ef8db9b04a851f", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.432833, + "spanId": "32dae310ea0aefba", + "parentSpanId": "3593b113e12132c7", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.873917, + "spanId": "3593b113e12132c7", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022334, + "spanId": "6f169c4f2dc4aaad", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.416917, + "spanId": "bb7746d5304c6484", + "parentSpanId": "b84d63f3c526e369", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.334959, + "spanId": "b84d63f3c526e369", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.030958, + "spanId": "0e18fd8c95595f59", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.0795, + "spanId": "1252b0ee3c28cf2c", + "parentSpanId": "a4e8a4433c76223a", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.567958, + "spanId": "a4e8a4433c76223a", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018, + "spanId": "6c3df87cb2548efc", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.840208, + "spanId": "b8f70a96f8582eac", + "parentSpanId": "ac22c12c9a505540", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.007917, + "spanId": "ac22c12c9a505540", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02325, + "spanId": "0ac8bd8629948fe5", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.838959, + "spanId": "cf473ff3dffb73a7", + "parentSpanId": "09921bcadac73c2a", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.123042, + "spanId": "09921bcadac73c2a", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.037125, + "spanId": "9ac602ebdae9a7c0", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.50525, + "spanId": "91bf32a23e0aa45d", + "parentSpanId": "278ce350087a3829", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.624292, + "spanId": "278ce350087a3829", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025583, + "spanId": "4061a6e7cf923b66", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.344792, + "spanId": "ecad7c97ec014990", + "parentSpanId": "908e3e0338f7f70f", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.189708, + "spanId": "908e3e0338f7f70f", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031083, + "spanId": "677e298ae4de8e13", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.288875, + "spanId": "637a95854d822f44", + "parentSpanId": "0e676ca1b4dedd2a", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.309625, + "spanId": "0e676ca1b4dedd2a", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024959, + "spanId": "3166fa6d419f70bb", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 19.025417, + "spanId": "3073930e396c5dd6", + "parentSpanId": "6189f8652a95dbf8", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.079833, + "spanId": "6189f8652a95dbf8", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.041333, + "spanId": "350ae3c8654130ec", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.069166, + "spanId": "2bb2835f409c139b", + "parentSpanId": "62740e1ebd8a8b81", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.14625, + "spanId": "62740e1ebd8a8b81", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + }, + { + "name": "skills.consume", + "durationMs": 308.208708, + "spanId": "830437259e669f47", + "parentSpanId": "e1056bda56df63b3", + "attributes": { + "paperclip.task.run.run_id": "2bbc759b0b1c" + } + } + ] + }, + { + "label": "warm-2", + "elapsedMs": 688.4290000000037, + "nativeAssetMs": 270.29879100000835, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1173569536, + "cpuUserMs": 370.926, + "cpuSystemMs": 532.599, + "processPeakRssBytes": 1173569536, + "rootPoolQueries": 5, + "traceId": "f6fc891515b3863e44ee0f8be7d8e771", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 89.272625, + "spanId": "060902a01806a14a", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.rows", + "durationMs": 50.614, + "spanId": "14c8f817ee1fde6d", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.035792, + "spanId": "9a6bca85685eaf68", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.488291, + "spanId": "f21743da07c91d7d", + "parentSpanId": "26ffb0ad2b85e6e6", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.310958, + "spanId": "26ffb0ad2b85e6e6", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.030459, + "spanId": "6509859f300acbfe", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.33175, + "spanId": "85d229e0cb148197", + "parentSpanId": "7108fb34e6c3ed34", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.86625, + "spanId": "7108fb34e6c3ed34", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023458, + "spanId": "95824a1ef4a3d153", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.349791, + "spanId": "ba310aafbbb73679", + "parentSpanId": "b682295241226103", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.602417, + "spanId": "b682295241226103", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021958, + "spanId": "425fee7e6203d54f", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 27.363166, + "spanId": "68da68d25bed9811", + "parentSpanId": "54a57a0d129dafc0", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 29.389833, + "spanId": "54a57a0d129dafc0", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03025, + "spanId": "85bdb32ac10acaee", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.543167, + "spanId": "48a19633a4c0483c", + "parentSpanId": "9644b4750599e760", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.61325, + "spanId": "9644b4750599e760", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.027083, + "spanId": "fca1aa43d12c5efc", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.091917, + "spanId": "e446186f7cf4f100", + "parentSpanId": "394da3c5e72c46f9", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.961083, + "spanId": "394da3c5e72c46f9", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03475, + "spanId": "51d13872e8be4418", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.925833, + "spanId": "681d908abdba4307", + "parentSpanId": "05254d42369095cc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.240583, + "spanId": "05254d42369095cc", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.027166, + "spanId": "6f211ed57a63ccb2", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.896083, + "spanId": "7ec61f46ce93a098", + "parentSpanId": "8b73850164882936", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.643042, + "spanId": "8b73850164882936", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.027, + "spanId": "71f03873e2bf8778", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.00825, + "spanId": "e0f506a91f64172f", + "parentSpanId": "5fb74ddde9b42aa5", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.666084, + "spanId": "5fb74ddde9b42aa5", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03925, + "spanId": "2e1f399f54e054ac", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.569, + "spanId": "c99b61fed9306b2d", + "parentSpanId": "91fc34a65b258c3b", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.538708, + "spanId": "91fc34a65b258c3b", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + }, + { + "name": "skills.consume", + "durationMs": 294.15025, + "spanId": "f4b6905adaf92377", + "parentSpanId": "5e61d0885481decc", + "attributes": { + "paperclip.task.run.run_id": "64125f5957f5" + } + } + ] + }, + { + "label": "warm-3", + "elapsedMs": 711.9416250000068, + "nativeAssetMs": 304.56324999999197, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1177747456, + "cpuUserMs": 372.507, + "cpuSystemMs": 504.664, + "processPeakRssBytes": 1177747456, + "rootPoolQueries": 5, + "traceId": "6d46021c20cbd4e6e961d993ff35d86c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 92.585625, + "spanId": "18960143afde921b", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.rows", + "durationMs": 24.387084, + "spanId": "57283c99547dabf5", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.032042, + "spanId": "fdab68018c787042", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.407, + "spanId": "06fb60fe0fac4f25", + "parentSpanId": "0c6389049282a1ed", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.067708, + "spanId": "0c6389049282a1ed", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021084, + "spanId": "f78000633e8a81b6", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.873209, + "spanId": "7d42cecaf908469e", + "parentSpanId": "1f176d4f358fa349", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.892042, + "spanId": "1f176d4f358fa349", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019, + "spanId": "0d14b2dda9c298b4", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.823291, + "spanId": "cce261badbb534db", + "parentSpanId": "6c0d699a5db65238", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.001208, + "spanId": "6c0d699a5db65238", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.072917, + "spanId": "0b9fb3f688af352d", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.441083, + "spanId": "2262aa2633ffb70d", + "parentSpanId": "60c3903778bd4541", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.273167, + "spanId": "60c3903778bd4541", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025375, + "spanId": "300cf843ae793b5a", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.061167, + "spanId": "5ef267c4b94045a4", + "parentSpanId": "581f429299383960", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.309542, + "spanId": "581f429299383960", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029584, + "spanId": "68225fe3a8272bee", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.676583, + "spanId": "5ec4f509a77f98be", + "parentSpanId": "a05becc03594b286", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.906875, + "spanId": "a05becc03594b286", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.043125, + "spanId": "6c17e2bf970a441c", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.083916, + "spanId": "4024235e52c185b5", + "parentSpanId": "0783e9ba182ad2d0", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 24.032209, + "spanId": "0783e9ba182ad2d0", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.044958, + "spanId": "40468e633b3f0863", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.144042, + "spanId": "3da9d7442ed346cb", + "parentSpanId": "4def2c65a95dc749", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.094542, + "spanId": "4def2c65a95dc749", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022208, + "spanId": "ca52d5cd6617c596", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.482292, + "spanId": "5dea864f4e97a5bd", + "parentSpanId": "26736b05eccf8260", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.9115, + "spanId": "26736b05eccf8260", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.046458, + "spanId": "b10c3280c1d60f44", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.391084, + "spanId": "29fbdde04442cb25", + "parentSpanId": "e6c16853ff1170b4", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.140959, + "spanId": "e6c16853ff1170b4", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + }, + { + "name": "skills.consume", + "durationMs": 327.715667, + "spanId": "dd0c896936d72f4b", + "parentSpanId": "0c5364167a5f2b16", + "attributes": { + "paperclip.task.run.run_id": "c36b6d25d99d" + } + } + ] + }, + { + "label": "warm-4", + "elapsedMs": 815.8250839999964, + "nativeAssetMs": 304.6120410000003, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1178189824, + "cpuUserMs": 391.625, + "cpuSystemMs": 554.741, + "processPeakRssBytes": 1178206208, + "rootPoolQueries": 5, + "traceId": "9f20b6f800b57fec38cb344d21fc6153", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 133.318542, + "spanId": "4613fa5f002ac318", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.rows", + "durationMs": 24.012959, + "spanId": "c94be52fa3505c23", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.039167, + "spanId": "61201cee5b056ad9", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.792167, + "spanId": "3aa72d77da9e4cdd", + "parentSpanId": "65159ed5d64df778", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.082667, + "spanId": "65159ed5d64df778", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031083, + "spanId": "3211c92366cbf0a1", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 23.405875, + "spanId": "8e6059475d0482d7", + "parentSpanId": "a1132748a4ceb117", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 27.072459, + "spanId": "a1132748a4ceb117", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.058667, + "spanId": "92c62687a08462e9", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.676208, + "spanId": "ad85f09e49538705", + "parentSpanId": "5e67cd60deb21440", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.205291, + "spanId": "5e67cd60deb21440", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031042, + "spanId": "babb8a3db6e51a71", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.413625, + "spanId": "eaad9fb2d366f9aa", + "parentSpanId": "65703ae92dbc1cff", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.003167, + "spanId": "65703ae92dbc1cff", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.041542, + "spanId": "967e915cd17dd81d", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.915667, + "spanId": "3bd57edb555c2f41", + "parentSpanId": "183119b9c6084300", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.244125, + "spanId": "183119b9c6084300", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.118833, + "spanId": "ab5f3ae30a1e0e68", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.589625, + "spanId": "91188a119576e5cd", + "parentSpanId": "988c15842263db37", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.700084, + "spanId": "988c15842263db37", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.043375, + "spanId": "4f3195aa6b439ada", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.826, + "spanId": "58637d350cab74eb", + "parentSpanId": "328bc410f5bd8c5c", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.670583, + "spanId": "328bc410f5bd8c5c", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0405, + "spanId": "3dd12efbe06671aa", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.727375, + "spanId": "ef323da3c3a16d57", + "parentSpanId": "5dd55a875f863fc4", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.299959, + "spanId": "5dd55a875f863fc4", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025334, + "spanId": "9331c9844ec3ecfd", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.875792, + "spanId": "d4eaad4a82125e36", + "parentSpanId": "3581cf04e567d7fc", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 62.447167, + "spanId": "3581cf04e567d7fc", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028625, + "spanId": "7c76e19e11df45cf", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.835459, + "spanId": "c0efe14f3c6afbf7", + "parentSpanId": "9762da0400e05ef6", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.268292, + "spanId": "9762da0400e05ef6", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + }, + { + "name": "skills.consume", + "durationMs": 327.780125, + "spanId": "b91ba87a9478fb2f", + "parentSpanId": "9fe81e92d645a7ce", + "attributes": { + "paperclip.task.run.run_id": "b8a78027be9e" + } + } + ] + }, + { + "label": "warm-5", + "elapsedMs": 723.6927079999878, + "nativeAssetMs": 281.0947080000042, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1181990912, + "cpuUserMs": 371.021, + "cpuSystemMs": 539.923, + "processPeakRssBytes": 1181990912, + "rootPoolQueries": 5, + "traceId": "2189e97eb2b25393001ffed519205068", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 91.356083, + "spanId": "d15c060883555679", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.rows", + "durationMs": 21.800792, + "spanId": "de58aa701a1a619c", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031125, + "spanId": "82972b8a1db50ca5", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.21825, + "spanId": "2a6879872a3ebba6", + "parentSpanId": "2f2090a815077c05", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.742084, + "spanId": "2f2090a815077c05", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021041, + "spanId": "0736fd8a38b586a8", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.880417, + "spanId": "5b7a7f5ec737b78d", + "parentSpanId": "9e37370a22c0f17a", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.337583, + "spanId": "9e37370a22c0f17a", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022042, + "spanId": "36362b73f345d107", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.485167, + "spanId": "40fe22ac7ee3c075", + "parentSpanId": "6b868a7405716b11", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.354125, + "spanId": "6b868a7405716b11", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021125, + "spanId": "0f5f5c5ce05fa71f", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.407125, + "spanId": "fa5b05b245447520", + "parentSpanId": "744dc935455d52dd", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.375458, + "spanId": "744dc935455d52dd", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023584, + "spanId": "9e67c269ffae6f3d", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.638958, + "spanId": "df125923d7c7a5c0", + "parentSpanId": "35553ee6e7609a6a", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.991875, + "spanId": "35553ee6e7609a6a", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026125, + "spanId": "8798f66101f1f69c", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.115541, + "spanId": "ddbc71a9ec29078c", + "parentSpanId": "35ce1079b6e3aefa", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.474416, + "spanId": "35ce1079b6e3aefa", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.035792, + "spanId": "33372845e5d9b915", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.943584, + "spanId": "cf445f9035cc12fa", + "parentSpanId": "135454c61abc83f7", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.460875, + "spanId": "135454c61abc83f7", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023041, + "spanId": "6d6bc6913a8143f9", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.297667, + "spanId": "3bc578f94789f967", + "parentSpanId": "ff668cad47ddbefa", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 51.456708, + "spanId": "ff668cad47ddbefa", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028708, + "spanId": "4c094a7e7ab86430", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.726291, + "spanId": "fa8e31d3286b7330", + "parentSpanId": "616cd86536b9ce60", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 25.449167, + "spanId": "616cd86536b9ce60", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021667, + "spanId": "61e9fc8861ae6b57", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.343333, + "spanId": "d5ab008d36ef2600", + "parentSpanId": "02ac698a9f091c03", + "attributes": { + "paperclip.task.run.run_id": "600626747605", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.328041, + "spanId": "02ac698a9f091c03", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + }, + { + "name": "skills.consume", + "durationMs": 305.00275, + "spanId": "3ffa09be181363e1", + "parentSpanId": "bdb83e528b17e93d", + "attributes": { + "paperclip.task.run.run_id": "600626747605" + } + } + ] + }, + { + "label": "warm-6", + "elapsedMs": 692.828874999992, + "nativeAssetMs": 298.1707499999902, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1185988608, + "cpuUserMs": 386.279, + "cpuSystemMs": 578.683, + "processPeakRssBytes": 1185988608, + "rootPoolQueries": 5, + "traceId": "b8234d9a319106e47acd03fbe548a2c7", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 94.715458, + "spanId": "aedc353c832ab3ab", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.rows", + "durationMs": 26.055334, + "spanId": "3f840a5f081fd28c", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.036166, + "spanId": "bc63d9f863f2359f", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.514542, + "spanId": "a9fde683037ebb86", + "parentSpanId": "832c37b87b14fd37", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.874292, + "spanId": "832c37b87b14fd37", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.035584, + "spanId": "d6c35cf9c58d99e4", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.130625, + "spanId": "79bffe885104fe59", + "parentSpanId": "4969ea6c07e47718", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.417875, + "spanId": "4969ea6c07e47718", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.051083, + "spanId": "13a8a683b0df6665", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.830375, + "spanId": "a028233a6771c2cd", + "parentSpanId": "ad6fe420926a494b", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.871208, + "spanId": "ad6fe420926a494b", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033958, + "spanId": "50696452911da230", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.58475, + "spanId": "6225b360b4bd611b", + "parentSpanId": "4a8892281b4ba2c2", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.851916, + "spanId": "4a8892281b4ba2c2", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024584, + "spanId": "1e5a5399ba790ee2", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.436708, + "spanId": "6bc08708fa7239f0", + "parentSpanId": "819c612b7985c660", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.753458, + "spanId": "819c612b7985c660", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023, + "spanId": "cd9e637a96b3dd86", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.652042, + "spanId": "778e4fd621b673ff", + "parentSpanId": "eb273bc455fb4601", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.898334, + "spanId": "eb273bc455fb4601", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025875, + "spanId": "556260aeaebcad0e", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.072375, + "spanId": "5137f757dccbcb6b", + "parentSpanId": "94e1f8753a5d56a6", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.372208, + "spanId": "94e1f8753a5d56a6", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029208, + "spanId": "cb57647f1825967b", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.132334, + "spanId": "c8db5698f4e3c096", + "parentSpanId": "f1a8f951083ca5d9", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.198208, + "spanId": "f1a8f951083ca5d9", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026833, + "spanId": "73856b6453b41419", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.94675, + "spanId": "d6516b0f78be05e2", + "parentSpanId": "18f0869166b06f40", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.020625, + "spanId": "18f0869166b06f40", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.050375, + "spanId": "8857d49eac48a844", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.131375, + "spanId": "76ac8ae85f39fc08", + "parentSpanId": "506cdd5913671263", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.399042, + "spanId": "506cdd5913671263", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + }, + { + "name": "skills.consume", + "durationMs": 322.229167, + "spanId": "69e972fed1225673", + "parentSpanId": "975e385637f2bcf3", + "attributes": { + "paperclip.task.run.run_id": "0e0b5fad526c" + } + } + ] + }, + { + "label": "warm-7", + "elapsedMs": 875.4426669999957, + "nativeAssetMs": 329.81108299999323, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1190395904, + "cpuUserMs": 392.125, + "cpuSystemMs": 650.12, + "processPeakRssBytes": 1190395904, + "rootPoolQueries": 5, + "traceId": "316101c00f4a73e281c068b814da6ebd", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 137.793208, + "spanId": "8983a1b297665486", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.rows", + "durationMs": 22.361834, + "spanId": "255b1aa3f8ee4398", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.071333, + "spanId": "c13a041b2e1ed54b", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.208625, + "spanId": "bfac1ef3a94b24e6", + "parentSpanId": "e4048d0e21b29698", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.335167, + "spanId": "e4048d0e21b29698", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.045416, + "spanId": "c61f5837a8dabf08", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.453167, + "spanId": "605cef8d689817c3", + "parentSpanId": "ae7f6a09d5bf281f", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.260542, + "spanId": "ae7f6a09d5bf281f", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.059209, + "spanId": "6c6404b76db640ce", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 24.3275, + "spanId": "a94395f15ab2d0e1", + "parentSpanId": "1ed17d32bf9beddc", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.069042, + "spanId": "1ed17d32bf9beddc", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020959, + "spanId": "2e7977fbc49ba76d", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 21.746166, + "spanId": "102e58245df91309", + "parentSpanId": "ef3cc8312d60f1f7", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 27.900084, + "spanId": "ef3cc8312d60f1f7", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026167, + "spanId": "e41fca6bc42bb793", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 24.673917, + "spanId": "2b9246acf3b3e46f", + "parentSpanId": "933697aa0968395b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.340042, + "spanId": "933697aa0968395b", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023208, + "spanId": "f3653a3a4d284e72", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 22.358917, + "spanId": "31ff787813b70581", + "parentSpanId": "67dc6bda595b8422", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.957584, + "spanId": "67dc6bda595b8422", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021583, + "spanId": "6b9a7f0cb5f08533", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 54.438042, + "spanId": "c167d29b2f2de2a0", + "parentSpanId": "30880fcbc23cba70", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 56.022125, + "spanId": "30880fcbc23cba70", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.070667, + "spanId": "3cae659df3dcd424", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 22.319375, + "spanId": "881eabb383fe5ba6", + "parentSpanId": "8ca12c644f1a168e", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 36.011667, + "spanId": "8ca12c644f1a168e", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028334, + "spanId": "41f55793e581ce51", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 25.668334, + "spanId": "fb162841b6c3b16a", + "parentSpanId": "39c5e0ab95d9cc43", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 27.801583, + "spanId": "39c5e0ab95d9cc43", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023875, + "spanId": "c29cee0e1be319e0", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.version.verify", + "durationMs": 18.3515, + "spanId": "7d719525f6b74a9b", + "parentSpanId": "ae75a5763a3b8e71", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.310375, + "spanId": "ae75a5763a3b8e71", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + }, + { + "name": "skills.consume", + "durationMs": 354.05125, + "spanId": "90b6e830cb412dd4", + "parentSpanId": "301c553a30fb5e9b", + "attributes": { + "paperclip.task.run.run_id": "bb77b8c75776" + } + } + ] + }, + { + "label": "warm-8", + "elapsedMs": 747.7878749999945, + "nativeAssetMs": 311.6350000000093, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1194590208, + "cpuUserMs": 377.021, + "cpuSystemMs": 801.387, + "processPeakRssBytes": 1194590208, + "rootPoolQueries": 5, + "traceId": "d97af7507b406a230f62012a303458d7", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.059, + "spanId": "e85ade283b58c3b6", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.rows", + "durationMs": 24.219583, + "spanId": "5b2a005ec4e6cc96", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.041125, + "spanId": "e91cfcfe26d4e206", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.435583, + "spanId": "ee1fa8de62f56baf", + "parentSpanId": "3511b3eeb3dba368", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.081292, + "spanId": "3511b3eeb3dba368", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019917, + "spanId": "0a7333292be748e4", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 18.895333, + "spanId": "66e0a5dac460e031", + "parentSpanId": "777b5794cb874b00", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 21.181709, + "spanId": "777b5794cb874b00", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018542, + "spanId": "d287c125aaaa6558", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.322875, + "spanId": "8bc9f5a3e7d319b0", + "parentSpanId": "cd8ab9ee8b47f0f0", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.875666, + "spanId": "cd8ab9ee8b47f0f0", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024959, + "spanId": "de095afc10fd80b2", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 19.542625, + "spanId": "e37758adb781fc96", + "parentSpanId": "cb15d30f17065a99", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 24.514459, + "spanId": "cb15d30f17065a99", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02375, + "spanId": "e21195dbdb2f475e", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.529583, + "spanId": "35032f3b8e2bece8", + "parentSpanId": "e461e99d3adb926b", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.925333, + "spanId": "e461e99d3adb926b", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019958, + "spanId": "a91cb11f65236054", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 18.435166, + "spanId": "8b4700bb388385d1", + "parentSpanId": "cb9e85e1397e3229", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.58525, + "spanId": "cb9e85e1397e3229", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031417, + "spanId": "e92616688b6c47ad", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 23.013458, + "spanId": "0a83e6a40c90f355", + "parentSpanId": "d05763dc69025db8", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 24.870125, + "spanId": "d05763dc69025db8", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017834, + "spanId": "6c79dd54c0457ac1", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 19.390833, + "spanId": "5e5d700631a4371a", + "parentSpanId": "7f024da6b0de0e17", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 21.392167, + "spanId": "7f024da6b0de0e17", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029667, + "spanId": "e5d5d425e699fa24", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.841541, + "spanId": "88b381ac1b85e212", + "parentSpanId": "f07c0ba0524ed071", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.474833, + "spanId": "f07c0ba0524ed071", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018875, + "spanId": "96573807066ec166", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.333583, + "spanId": "c2a35f6b9180ab40", + "parentSpanId": "f435eb1091231cb6", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.223542, + "spanId": "f435eb1091231cb6", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + }, + { + "name": "skills.consume", + "durationMs": 334.012625, + "spanId": "ae008dbf63f70329", + "parentSpanId": "a4c9c33888654237", + "attributes": { + "paperclip.task.run.run_id": "39a29de17032" + } + } + ] + }, + { + "label": "warm-9", + "elapsedMs": 671.4019580000022, + "nativeAssetMs": 283.660916000008, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196359680, + "cpuUserMs": 380.548, + "cpuSystemMs": 663.329, + "processPeakRssBytes": 1196359680, + "rootPoolQueries": 5, + "traceId": "158c21552f85208b7e9205746855cdd1", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 93.567416, + "spanId": "0a154252dd392df2", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.rows", + "durationMs": 24.245459, + "spanId": "b9bdfe482150bf1f", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.037667, + "spanId": "a9cba88446216f7b", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.801667, + "spanId": "581512d6a85f27bc", + "parentSpanId": "155c9905c1352bf3", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.220375, + "spanId": "155c9905c1352bf3", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029042, + "spanId": "f6afce507e8571df", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.729458, + "spanId": "d5261044154b7f4d", + "parentSpanId": "ad653bb27ffafca4", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 21.094083, + "spanId": "ad653bb27ffafca4", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020416, + "spanId": "9f9623ba8226000f", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.672042, + "spanId": "719edc30dd8156dd", + "parentSpanId": "ad1ad65946448006", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.543625, + "spanId": "ad1ad65946448006", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.029666, + "spanId": "073c115f656e8b77", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.733041, + "spanId": "e4f0b206dcfa056a", + "parentSpanId": "3f3d03c21cd5ca86", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.21625, + "spanId": "3f3d03c21cd5ca86", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0215, + "spanId": "ff107a15d448b0a3", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.275625, + "spanId": "d4b4122de2e1053e", + "parentSpanId": "c996382893a4d33d", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.548458, + "spanId": "c996382893a4d33d", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017083, + "spanId": "b8c87b25b2c138a9", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.757958, + "spanId": "40931943da3fdf4d", + "parentSpanId": "e05ec6ad3f3d09df", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.661125, + "spanId": "e05ec6ad3f3d09df", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017167, + "spanId": "0c0a60e224ebbc29", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.045417, + "spanId": "f1905953fba53348", + "parentSpanId": "f8823bc0909432a9", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.995375, + "spanId": "f8823bc0909432a9", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022625, + "spanId": "c0b314908ce4314f", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.8535, + "spanId": "6476499c1378babc", + "parentSpanId": "a169be093b96c12d", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.802417, + "spanId": "a169be093b96c12d", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016792, + "spanId": "c00e6fa26d4e477e", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.57775, + "spanId": "ce3e66f87abcd5af", + "parentSpanId": "28766b527b03d680", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.794083, + "spanId": "28766b527b03d680", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0185, + "spanId": "1725313dbc38cb3f", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.994292, + "spanId": "d1fe27e6f0fb88c2", + "parentSpanId": "852b533fc3d1c41d", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.337375, + "spanId": "852b533fc3d1c41d", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + }, + { + "name": "skills.consume", + "durationMs": 307.7825, + "spanId": "525e896569b02a35", + "parentSpanId": "36c0938527bed6ce", + "attributes": { + "paperclip.task.run.run_id": "7fea0c7d5620" + } + } + ] + }, + { + "label": "warm-10", + "elapsedMs": 668.2096250000031, + "nativeAssetMs": 277.69979199999943, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196769280, + "cpuUserMs": 371.548, + "cpuSystemMs": 562.298, + "processPeakRssBytes": 1196769280, + "rootPoolQueries": 5, + "traceId": "9c3c4b127e6fbf0fcbda0a3291463287", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 98.237833, + "spanId": "fb7a743232ce9d84", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.rows", + "durationMs": 23.887291, + "spanId": "003ef8acec63e9aa", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.040625, + "spanId": "d19c5395c411bb1c", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.5185, + "spanId": "c587d889916fef44", + "parentSpanId": "1422e5cd1276f48b", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.843208, + "spanId": "1422e5cd1276f48b", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01825, + "spanId": "d710fded7e41266e", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.854125, + "spanId": "f3952231be884e32", + "parentSpanId": "405056cd329b2d1e", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.708834, + "spanId": "405056cd329b2d1e", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017458, + "spanId": "6fe40fb9ccbb0c49", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.518834, + "spanId": "0d90499ee67ac369", + "parentSpanId": "11139fe4d944c2d9", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 27.902875, + "spanId": "11139fe4d944c2d9", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017875, + "spanId": "aeef75b4a3396dc6", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.197625, + "spanId": "20daa90f77d2a9ea", + "parentSpanId": "1b7a9a50b414387a", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.225542, + "spanId": "1b7a9a50b414387a", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018708, + "spanId": "5a799fe73cff2033", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.3595, + "spanId": "337b84d6c76fb414", + "parentSpanId": "879f84c961c65633", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.210291, + "spanId": "879f84c961c65633", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.047541, + "spanId": "7650913d055ec95d", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.410417, + "spanId": "c9fa4a4600d02fc3", + "parentSpanId": "470a4a902b7de434", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.467208, + "spanId": "470a4a902b7de434", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023291, + "spanId": "925c4d46895a8e09", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.145084, + "spanId": "fcafe69563f6ea82", + "parentSpanId": "2e2bcbda55e1ef79", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.202416, + "spanId": "2e2bcbda55e1ef79", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021833, + "spanId": "1b7ef7a7c0a09c11", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.990958, + "spanId": "a8913945898c0046", + "parentSpanId": "ad78fdeffab5b633", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.396667, + "spanId": "ad78fdeffab5b633", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0325, + "spanId": "5691e4ea778dec6d", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.330709, + "spanId": "7186a056af575152", + "parentSpanId": "0c5cbfe75b1be084", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.621417, + "spanId": "0c5cbfe75b1be084", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025792, + "spanId": "f0538f6403c45779", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.0955, + "spanId": "836cadcb20dbc05b", + "parentSpanId": "a8f980e4360b04da", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.268583, + "spanId": "a8f980e4360b04da", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + }, + { + "name": "skills.consume", + "durationMs": 301.668459, + "spanId": "49f584232f6a960b", + "parentSpanId": "89b234788e9718db", + "attributes": { + "paperclip.task.run.run_id": "73e2d018b945" + } + } + ] + }, + { + "label": "warm-11", + "elapsedMs": 672.346458, + "nativeAssetMs": 285.53370800000266, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196769280, + "cpuUserMs": 358.553, + "cpuSystemMs": 552.457, + "processPeakRssBytes": 1196769280, + "rootPoolQueries": 5, + "traceId": "022c6b37377415db1545b3c051352546", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 94.570666, + "spanId": "cc56babdbd788833", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.rows", + "durationMs": 23.555584, + "spanId": "909eb1818b1eb542", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.122708, + "spanId": "f57ecdf5b3ee7f92", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.567625, + "spanId": "5eeee09f44a72910", + "parentSpanId": "f46fb8b9a3ab5bf8", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 40.547333, + "spanId": "f46fb8b9a3ab5bf8", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017, + "spanId": "48330e3aa4907abd", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.450166, + "spanId": "74b594fcc52334b0", + "parentSpanId": "4546edcd47c7d4cd", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.484375, + "spanId": "4546edcd47c7d4cd", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01925, + "spanId": "7cffb65bf3732f88", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.773791, + "spanId": "7ca38aea3f39c353", + "parentSpanId": "838e542666b6ea70", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.954625, + "spanId": "838e542666b6ea70", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026625, + "spanId": "8f8d6365132c6c79", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.178542, + "spanId": "0736007e870ff3f6", + "parentSpanId": "3b20728e82bd1475", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.188125, + "spanId": "3b20728e82bd1475", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.026292, + "spanId": "40f790790e67a847", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.658959, + "spanId": "848108ce9b39120c", + "parentSpanId": "3a7d0edecbd2ebc2", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.386625, + "spanId": "3a7d0edecbd2ebc2", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016084, + "spanId": "88c0952332d27798", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.141416, + "spanId": "8823f050996424d0", + "parentSpanId": "b8363f9a96aa799a", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.486459, + "spanId": "b8363f9a96aa799a", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0135, + "spanId": "33abe9d71139acf5", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.473084, + "spanId": "f0126683d42010da", + "parentSpanId": "dff8e4078e5d0680", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.226791, + "spanId": "dff8e4078e5d0680", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013208, + "spanId": "34af429e3bd5af36", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.648833, + "spanId": "9134dfd32cae1c90", + "parentSpanId": "e520ad331c98aae6", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.264625, + "spanId": "e520ad331c98aae6", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020792, + "spanId": "c9cbd05cda0f7176", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.428833, + "spanId": "8440bc51427b378f", + "parentSpanId": "9e29c65d09b830ed", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.277541, + "spanId": "9e29c65d09b830ed", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.021, + "spanId": "fd25067abff5ee02", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.303125, + "spanId": "2a43ff36181534cd", + "parentSpanId": "bb6c030e8ae7b6e2", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.08825, + "spanId": "bb6c030e8ae7b6e2", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + }, + { + "name": "skills.consume", + "durationMs": 309.249375, + "spanId": "27139f6dc36c1c0a", + "parentSpanId": "b7f44748253d1b65", + "attributes": { + "paperclip.task.run.run_id": "8e0e06796252" + } + } + ] + }, + { + "label": "warm-12", + "elapsedMs": 723.1867499999935, + "nativeAssetMs": 311.0507079999952, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196785664, + "cpuUserMs": 387.377, + "cpuSystemMs": 567.72, + "processPeakRssBytes": 1196785664, + "rootPoolQueries": 5, + "traceId": "8dfbffb029157b820792369a471e6950", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 95.063042, + "spanId": "3637e24cc45eb6f8", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.rows", + "durationMs": 24.186, + "spanId": "04c5e857670a6577", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031833, + "spanId": "e2c84cfbdafc16be", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 25.678625, + "spanId": "ea1ec70180d9e2be", + "parentSpanId": "5b6e2ac5c61f4192", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 31.75025, + "spanId": "5b6e2ac5c61f4192", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03225, + "spanId": "64447a9d7423c531", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.461708, + "spanId": "886ddcb08ceb6d24", + "parentSpanId": "959da2cb8987c28f", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.344541, + "spanId": "959da2cb8987c28f", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0155, + "spanId": "63e8574e224b555a", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.926041, + "spanId": "59e409a06edfefc5", + "parentSpanId": "4679b8efd0317998", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.536375, + "spanId": "4679b8efd0317998", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02025, + "spanId": "6df0eec22ff2a65e", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.600417, + "spanId": "a7fb66554d618bc9", + "parentSpanId": "0acc11ae45d6e766", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.112916, + "spanId": "0acc11ae45d6e766", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025375, + "spanId": "9b55c3e3c28c309a", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.815417, + "spanId": "3d6a2a276d37aa9c", + "parentSpanId": "ad326efafecb2727", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.220583, + "spanId": "ad326efafecb2727", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019959, + "spanId": "8b2e8b9220704cf0", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.458833, + "spanId": "2584e7e442577968", + "parentSpanId": "844781a1b3f9415e", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.14875, + "spanId": "844781a1b3f9415e", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023833, + "spanId": "ea257174968f741a", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.802917, + "spanId": "9a03a17253daa3ed", + "parentSpanId": "f687dd834f6079db", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.741791, + "spanId": "f687dd834f6079db", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020167, + "spanId": "e6b8a4564d342577", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.60725, + "spanId": "0c643f6f7f01f583", + "parentSpanId": "76da6221cbae5abb", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.825, + "spanId": "76da6221cbae5abb", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.070375, + "spanId": "d03955486894a051", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.219666, + "spanId": "af793ebcf23992b6", + "parentSpanId": "d0c49f702252f68c", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.298292, + "spanId": "d0c49f702252f68c", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015959, + "spanId": "57ee0f4ae2906905", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.218291, + "spanId": "e93b5674b3f692dd", + "parentSpanId": "ffc2b16ae38cc872", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.2255, + "spanId": "ffc2b16ae38cc872", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + }, + { + "name": "skills.consume", + "durationMs": 334.045792, + "spanId": "5674cdb81ce53f15", + "parentSpanId": "68ed90d5f5e1e077", + "attributes": { + "paperclip.task.run.run_id": "8b171869f93e" + } + } + ] + }, + { + "label": "warm-13", + "elapsedMs": 792.8505829999922, + "nativeAssetMs": 345.88183300000674, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1196851200, + "cpuUserMs": 398.725, + "cpuSystemMs": 540.204, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "ab0e83463bc450d0d50b4c69ad1e11e9", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 96.953, + "spanId": "7c071cb28e7358dd", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.rows", + "durationMs": 24.012458, + "spanId": "55bc10a5898a0f63", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033459, + "spanId": "1b30fc4efa3c514e", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 19.479875, + "spanId": "4dacde57a9f4ffa6", + "parentSpanId": "5764f9f070fd5427", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.039208, + "spanId": "5764f9f070fd5427", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020791, + "spanId": "96bd7c924d37a4d5", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.505, + "spanId": "e20a0ca38ae05860", + "parentSpanId": "6d88788e06d2cded", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 29.035, + "spanId": "6d88788e06d2cded", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019667, + "spanId": "d38a245da09258ae", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.77, + "spanId": "710819766b964ec0", + "parentSpanId": "f846e8a5a57c6163", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.220208, + "spanId": "f846e8a5a57c6163", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.089, + "spanId": "0b85226d0792714e", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.778542, + "spanId": "d827795876f25b55", + "parentSpanId": "9475b0ec08321af7", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 30.210333, + "spanId": "9475b0ec08321af7", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.027417, + "spanId": "4f9d9e62f4dc4648", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 17.429, + "spanId": "5d9e36db871ea0bd", + "parentSpanId": "872d044bf3336b3d", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 23.475542, + "spanId": "872d044bf3336b3d", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.042167, + "spanId": "5c38a59db88266cb", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 18.609291, + "spanId": "233b6d2849326f87", + "parentSpanId": "3620704c0d6902df", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 21.441583, + "spanId": "3620704c0d6902df", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.037083, + "spanId": "db7d1d6aa1bbac5e", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.889792, + "spanId": "7bbfc1b7443b3d42", + "parentSpanId": "4e59c150960b43e5", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.036917, + "spanId": "4e59c150960b43e5", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018, + "spanId": "0cda4055390cd6df", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.488583, + "spanId": "cc4f3e43e6bff564", + "parentSpanId": "e59fdf7193dec64d", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.103583, + "spanId": "e59fdf7193dec64d", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019333, + "spanId": "a8edd2ed1679a177", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.51625, + "spanId": "09d369cb0890c7bb", + "parentSpanId": "2f86ee841ee523f4", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.863791, + "spanId": "2f86ee841ee523f4", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020416, + "spanId": "b85bb2aa450ba319", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.116333, + "spanId": "a603978033894295", + "parentSpanId": "ae2c40acf31050f3", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.185, + "spanId": "ae2c40acf31050f3", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + }, + { + "name": "skills.consume", + "durationMs": 370.029, + "spanId": "aeffbdcade2d443a", + "parentSpanId": "b38814b9d700aca0", + "attributes": { + "paperclip.task.run.run_id": "1b35ab2142aa" + } + } + ] + }, + { + "label": "warm-14", + "elapsedMs": 752.5533330000035, + "nativeAssetMs": 293.2099579999922, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1169276928, + "cpuUserMs": 380.094, + "cpuSystemMs": 604.486, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "c19c6ba80aece473d3eca1071182f6af", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 96.714458, + "spanId": "a505ecd060a814f5", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.rows", + "durationMs": 23.460459, + "spanId": "cbe82a934258a363", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.034833, + "spanId": "1d7a20b0d88dfa07", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.483459, + "spanId": "c9386cc08189b5b2", + "parentSpanId": "06eb9689f0428064", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.7495, + "spanId": "06eb9689f0428064", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017834, + "spanId": "ca5fc3a38e3d5f6a", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 16.10375, + "spanId": "8e19020c784ebcc3", + "parentSpanId": "33acfedfb80c46cb", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 18.613917, + "spanId": "33acfedfb80c46cb", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017084, + "spanId": "4c08eedb3a58aa33", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.501209, + "spanId": "25e06bf2c14e5eff", + "parentSpanId": "95dfc2a9e497f11a", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.521375, + "spanId": "95dfc2a9e497f11a", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017791, + "spanId": "5d3c3963206290f4", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.547459, + "spanId": "cb352b722199d5e2", + "parentSpanId": "354de89589f230c5", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.6525, + "spanId": "354de89589f230c5", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019375, + "spanId": "bd4fa4aa651a056e", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.349834, + "spanId": "d2b551d170b127b1", + "parentSpanId": "4139293c0a84d3a6", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.434083, + "spanId": "4139293c0a84d3a6", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016958, + "spanId": "f0ab419d2e2a0664", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.505833, + "spanId": "2c7a04272875220c", + "parentSpanId": "c5f742ee063de31b", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.832792, + "spanId": "c5f742ee063de31b", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017584, + "spanId": "0089a417ef6549d1", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 15.226334, + "spanId": "52867b3fb96547ae", + "parentSpanId": "4c0ffb3a3e1178bd", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 26.4505, + "spanId": "4c0ffb3a3e1178bd", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024084, + "spanId": "2bb307893398218b", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 21.191625, + "spanId": "06a235e04a8cd431", + "parentSpanId": "9e393715b4f2cd74", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 69.550542, + "spanId": "9e393715b4f2cd74", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.019541, + "spanId": "de389ef30eb65ef2", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.9225, + "spanId": "5c0b58d67ea27a19", + "parentSpanId": "2196e41e087c3f93", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.361041, + "spanId": "2196e41e087c3f93", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018416, + "spanId": "bc5f20d4507a68ed", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.256625, + "spanId": "8b923960f779829f", + "parentSpanId": "68018e98e4297be6", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.004375, + "spanId": "68018e98e4297be6", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + }, + { + "name": "skills.consume", + "durationMs": 317.462292, + "spanId": "5e7be438767e9c5f", + "parentSpanId": "72f721e77fda3a3f", + "attributes": { + "paperclip.task.run.run_id": "7f68984f9e9e" + } + } + ] + }, + { + "label": "warm-15", + "elapsedMs": 678.2098330000008, + "nativeAssetMs": 292.1170829999901, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170112512, + "cpuUserMs": 368.551, + "cpuSystemMs": 601.134, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "3b93d864814cb1d637c2a3ee4fc8f7ee", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 95.106333, + "spanId": "f0ed73d3050e9260", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.rows", + "durationMs": 22.830958, + "spanId": "e9159aff074939a1", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.030667, + "spanId": "39b0cfaab1ecbb00", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.606041, + "spanId": "628ca4f20dee0e3f", + "parentSpanId": "08e5af25f48d60c9", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 20.862834, + "spanId": "08e5af25f48d60c9", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017333, + "spanId": "0b0ef4b89eab4f41", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.524166, + "spanId": "e4ce400512994b1f", + "parentSpanId": "8ea7d402824d6a95", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 19.716958, + "spanId": "8ea7d402824d6a95", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.036041, + "spanId": "17a9b85b1d219230", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.863583, + "spanId": "37898c03b730c413", + "parentSpanId": "b13c92d9e6ae479f", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.674708, + "spanId": "b13c92d9e6ae479f", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018875, + "spanId": "3b5869683f013175", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.112709, + "spanId": "4df180f3ac88ac05", + "parentSpanId": "d1b465639e6d2299", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.175416, + "spanId": "d1b465639e6d2299", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017083, + "spanId": "8dcbe861fd91a538", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.086042, + "spanId": "a2a2bba6e9a29ee5", + "parentSpanId": "48781e504377a7f0", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.219375, + "spanId": "48781e504377a7f0", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017541, + "spanId": "25509da5b6495d0d", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.891375, + "spanId": "f5bfd7f7bf8f1d2b", + "parentSpanId": "db6850b5ccce8550", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.721125, + "spanId": "db6850b5ccce8550", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.033458, + "spanId": "f87ebc9e1f5ec255", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.967584, + "spanId": "d2b6db2b3362ee72", + "parentSpanId": "6ff618f2d8a2af30", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.916875, + "spanId": "6ff618f2d8a2af30", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022792, + "spanId": "12940b6b35524eb4", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 14.824041, + "spanId": "3b988ad31631a66a", + "parentSpanId": "e29bdd7f785ab7da", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 17.072959, + "spanId": "e29bdd7f785ab7da", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.022167, + "spanId": "095c039d99fb0b49", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.191459, + "spanId": "44fd7ada1602c23a", + "parentSpanId": "35ab9a95079410c0", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.235167, + "spanId": "35ab9a95079410c0", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02175, + "spanId": "167d297d6daec949", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.836541, + "spanId": "65563c8a32a1ee58", + "parentSpanId": "6dc5d5f206c06612", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.779041, + "spanId": "6dc5d5f206c06612", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + }, + { + "name": "skills.consume", + "durationMs": 314.572792, + "spanId": "8ff40b67f4d4f910", + "parentSpanId": "98a462fd9b830a94", + "attributes": { + "paperclip.task.run.run_id": "c49babd099dd" + } + } + ] + }, + { + "label": "warm-16", + "elapsedMs": 666.0508750000008, + "nativeAssetMs": 289.2593749999942, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170112512, + "cpuUserMs": 352.025, + "cpuSystemMs": 574.008, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "d227d3ecd0965bb710050923c10cb8fc", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 89.368875, + "spanId": "5f3f0bdfb48cbef9", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.rows", + "durationMs": 21.964416, + "spanId": "eeb2387367889517", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031917, + "spanId": "27fd3661a847fbd0", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.822791, + "spanId": "726648458f75d9b7", + "parentSpanId": "6766f1d0fed61f82", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.922, + "spanId": "6766f1d0fed61f82", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.05275, + "spanId": "9f5816fba6b55d4d", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.012792, + "spanId": "f847ef4c717eccb4", + "parentSpanId": "42986c651cbe40e8", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 28.399708, + "spanId": "42986c651cbe40e8", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016375, + "spanId": "242d44665082fbb7", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.813042, + "spanId": "7461be53148cfb26", + "parentSpanId": "8c196cb1b2b5f9ee", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.390708, + "spanId": "8c196cb1b2b5f9ee", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016167, + "spanId": "26061f4b061b03ed", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.804459, + "spanId": "5ec473613d1b210a", + "parentSpanId": "f13db5269b06f060", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.613958, + "spanId": "f13db5269b06f060", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.039, + "spanId": "2525e86ec6b917be", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.988917, + "spanId": "66ce3f7a716e9c5b", + "parentSpanId": "03cf963c27ba5e1b", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.177667, + "spanId": "03cf963c27ba5e1b", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01825, + "spanId": "04dd3c050cb57af1", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.381291, + "spanId": "dadc901e937b3c08", + "parentSpanId": "5faa408d4ba3ee0a", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.156125, + "spanId": "5faa408d4ba3ee0a", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015125, + "spanId": "d9f8803fb8ecaf1b", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.229, + "spanId": "552e76c6ae1af094", + "parentSpanId": "f967a1b6eb4ad0a9", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.78275, + "spanId": "f967a1b6eb4ad0a9", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018583, + "spanId": "8a3eee52f009a428", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.269917, + "spanId": "8cdc7e8455100713", + "parentSpanId": "0675393ed1dfa077", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.112542, + "spanId": "0675393ed1dfa077", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01575, + "spanId": "f8d579620d5dcb52", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.170042, + "spanId": "3c1caeca4b857bc7", + "parentSpanId": "f1fbce3d84b6b93f", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.001084, + "spanId": "f1fbce3d84b6b93f", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03325, + "spanId": "bdcd4d658d256acb", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.165125, + "spanId": "2f0d46b5e557e7c2", + "parentSpanId": "8a74b68966637939", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.528375, + "spanId": "8a74b68966637939", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + }, + { + "name": "skills.consume", + "durationMs": 310.610583, + "spanId": "2eaff2f06a8504b8", + "parentSpanId": "bd9a8ee2ed56981d", + "attributes": { + "paperclip.task.run.run_id": "20e869d67494" + } + } + ] + }, + { + "label": "warm-17", + "elapsedMs": 629.3935419999907, + "nativeAssetMs": 265.11266699999396, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170178048, + "cpuUserMs": 342.554, + "cpuSystemMs": 539.634, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "0959ca2eb377863360c991b424da0f13", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.912875, + "spanId": "71c9a03dafcde650", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.rows", + "durationMs": 22.80825, + "spanId": "803a3f22bf442f71", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.0285, + "spanId": "ebb5e4a7e13b1369", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.10625, + "spanId": "fef12f81d694e925", + "parentSpanId": "e3e411a49b9a6d4e", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.957208, + "spanId": "e3e411a49b9a6d4e", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.03675, + "spanId": "57a4eafff7f48155", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.771167, + "spanId": "f36ee51d0d65dfcc", + "parentSpanId": "a9bfba0d0c1914c4", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.234625, + "spanId": "a9bfba0d0c1914c4", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.043125, + "spanId": "b86350c4b6e2c429", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.41425, + "spanId": "4ed5ea96531cd339", + "parentSpanId": "02d32218dcf1d2e5", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.883125, + "spanId": "02d32218dcf1d2e5", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020625, + "spanId": "bf2b3b7520d11797", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.661042, + "spanId": "f70f9849e221f567", + "parentSpanId": "510fef9f53405cb5", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.160041, + "spanId": "510fef9f53405cb5", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01325, + "spanId": "aba98a7709122283", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.827708, + "spanId": "baa65af8721c1b92", + "parentSpanId": "db31fcfc11e8307f", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 24.692791, + "spanId": "db31fcfc11e8307f", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015208, + "spanId": "f65a29890fe31116", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.929291, + "spanId": "e812810129fc4914", + "parentSpanId": "ebc6c09b543bdef1", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.221125, + "spanId": "ebc6c09b543bdef1", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013541, + "spanId": "85eb3477665173f2", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.543084, + "spanId": "5cdf9cf3c2101660", + "parentSpanId": "8cf973386c605384", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.558708, + "spanId": "8cf973386c605384", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.040166, + "spanId": "827b53fcc7d60a7e", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.980125, + "spanId": "76b1056aaf7238a5", + "parentSpanId": "6ccb285228518d06", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.877833, + "spanId": "6ccb285228518d06", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.043459, + "spanId": "adb39cd953ef889d", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.473333, + "spanId": "a3a54e035cc038f2", + "parentSpanId": "d91a62ca7d452bf3", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.260083, + "spanId": "d91a62ca7d452bf3", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028625, + "spanId": "4785cf6f969071a8", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.676208, + "spanId": "384b2b4d06ee9288", + "parentSpanId": "6cb6f53c6c887684", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.268417, + "spanId": "6cb6f53c6c887684", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + }, + { + "name": "skills.consume", + "durationMs": 287.488583, + "spanId": "230e657720325807", + "parentSpanId": "4d02a6ee99cdb6e0", + "attributes": { + "paperclip.task.run.run_id": "19bdba458719" + } + } + ] + }, + { + "label": "warm-18", + "elapsedMs": 587.9979170000006, + "nativeAssetMs": 266.7930420000048, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170194432, + "cpuUserMs": 323.898, + "cpuSystemMs": 502.382, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "7add063fd46cd3f784136c6c1774c933", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 85.881917, + "spanId": "17523936b48291e6", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.rows", + "durationMs": 23.638917, + "spanId": "efbebe2b7783a4c9", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.025292, + "spanId": "3cfb8377a8ed5149", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.930458, + "spanId": "593ac73ea679a6d4", + "parentSpanId": "52119f530cafe0e8", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.315167, + "spanId": "52119f530cafe0e8", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014792, + "spanId": "59225536d3d6cbe4", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.476459, + "spanId": "c126080b47be7724", + "parentSpanId": "94b7343b63d76f45", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.521333, + "spanId": "94b7343b63d76f45", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.012458, + "spanId": "272cc0bd62f9fd1b", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.334125, + "spanId": "c5a3a1caf17e4f53", + "parentSpanId": "4708c01925f1e494", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.398791, + "spanId": "4708c01925f1e494", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.012083, + "spanId": "99c4295c2da6c50d", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.18875, + "spanId": "ebf451a41a0c342a", + "parentSpanId": "5af925b154caaf30", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.28725, + "spanId": "5af925b154caaf30", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013792, + "spanId": "39bb8dd7233e6de7", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.638792, + "spanId": "816b0c3449805d30", + "parentSpanId": "446e46ba9ff2a2fe", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.516792, + "spanId": "446e46ba9ff2a2fe", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014625, + "spanId": "501c1fcf0c0ca3f4", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.474625, + "spanId": "32eb967849144dda", + "parentSpanId": "83c56970fac108ca", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 9.49525, + "spanId": "83c56970fac108ca", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013, + "spanId": "ccfeeb6e5c0d5d2f", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.691084, + "spanId": "1d1ff3fc2c7350b5", + "parentSpanId": "3094093d4bf3570e", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.010166, + "spanId": "3094093d4bf3570e", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013417, + "spanId": "f645cff1e9e7b7bf", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.088333, + "spanId": "92a8a3ae1618daeb", + "parentSpanId": "2f65e778eac62cb5", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.200708, + "spanId": "2f65e778eac62cb5", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.02975, + "spanId": "c10bac3df4223c75", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.962708, + "spanId": "7ec439ede98d8b07", + "parentSpanId": "792562ab10e88b07", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.095375, + "spanId": "792562ab10e88b07", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015459, + "spanId": "b8797e7a20aa91be", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.073875, + "spanId": "397691258e7ada02", + "parentSpanId": "fe1c5ed3b4b8b259", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.463334, + "spanId": "fe1c5ed3b4b8b259", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + }, + { + "name": "skills.consume", + "durationMs": 289.033667, + "spanId": "f9aafc385d9a93e8", + "parentSpanId": "db32579095f0635a", + "attributes": { + "paperclip.task.run.run_id": "12b5960330ee" + } + } + ] + }, + { + "label": "warm-19", + "elapsedMs": 559.9573340000061, + "nativeAssetMs": 234.43945900000108, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1170194432, + "cpuUserMs": 295.048, + "cpuSystemMs": 492.825, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "369dbe7d6a967e11de29815c49a8ad5c", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 86.37125, + "spanId": "f88b9f8385f8dda6", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.rows", + "durationMs": 23.303958, + "spanId": "a399c47cacc9cc9e", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.024958, + "spanId": "ce3f7468d95e92d0", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.060584, + "spanId": "ad1b99c3023c526b", + "parentSpanId": "7debd6f6cb31e3ba", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.376916, + "spanId": "7debd6f6cb31e3ba", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01775, + "spanId": "d873cf09b3d101bb", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.433333, + "spanId": "7c4d5c3b3ddab135", + "parentSpanId": "3d3da8cb9b92ebcb", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.509167, + "spanId": "3d3da8cb9b92ebcb", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013125, + "spanId": "3b55d90f6b3e02ed", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.914791, + "spanId": "df0b9fa1de262d09", + "parentSpanId": "da65185486325019", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 9.796666, + "spanId": "da65185486325019", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013667, + "spanId": "b06b8f24383889bb", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 8.97275, + "spanId": "ce3f59e14d660abb", + "parentSpanId": "62055cb41f4c3f9d", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.699666, + "spanId": "62055cb41f4c3f9d", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.028041, + "spanId": "ca4e6075f2c156fe", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.249209, + "spanId": "d3550ddb45490add", + "parentSpanId": "3c3af017fcc62998", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.580708, + "spanId": "3c3af017fcc62998", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014625, + "spanId": "32bc17825cb90cbe", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.035875, + "spanId": "a7a080f77ac6dbc3", + "parentSpanId": "84e5273d7328d97e", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.557125, + "spanId": "84e5273d7328d97e", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013042, + "spanId": "1ab8b3702b9cae58", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.294125, + "spanId": "3f84c407d156523b", + "parentSpanId": "e70077955191232f", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 10.566584, + "spanId": "e70077955191232f", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017959, + "spanId": "8cbe04050537068b", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.515125, + "spanId": "df2777fa0857f7ed", + "parentSpanId": "c58f8979a201ce46", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.866791, + "spanId": "c58f8979a201ce46", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016583, + "spanId": "2861c402508b345f", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.939834, + "spanId": "815c6c0fcaa9050b", + "parentSpanId": "c76aff3d30d3dd6b", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.424084, + "spanId": "c76aff3d30d3dd6b", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015833, + "spanId": "9d072a57fac39dcf", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.1085, + "spanId": "1eafb0178f2e69dc", + "parentSpanId": "6a70ca177da1a41c", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.468583, + "spanId": "6a70ca177da1a41c", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + }, + { + "name": "skills.consume", + "durationMs": 257.566333, + "spanId": "6c0bc7cd5dd496fd", + "parentSpanId": "680f4a019e0d7301", + "attributes": { + "paperclip.task.run.run_id": "375aba6ab7ce" + } + } + ] + }, + { + "label": "warm-20", + "elapsedMs": 595.2773749999906, + "nativeAssetMs": 244.01191699999617, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1171095552, + "cpuUserMs": 315.163, + "cpuSystemMs": 504.002, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "02efd58077f4cfcd7b4843d365ae2115", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 89.567, + "spanId": "4f637be809c56234", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.rows", + "durationMs": 23.273334, + "spanId": "f356f04c031a3d04", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.023958, + "spanId": "9a20808ed5bd9b83", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.060166, + "spanId": "bab8c6d90715408e", + "parentSpanId": "f5740b94daf6f691", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.699541, + "spanId": "f5740b94daf6f691", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017708, + "spanId": "36f545ca1e08c604", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.15025, + "spanId": "5805c18d2b0505d6", + "parentSpanId": "f329845811fc2f89", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.08375, + "spanId": "f329845811fc2f89", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.031333, + "spanId": "86d67668e78f56e5", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.966, + "spanId": "af57aa15ab8a3c20", + "parentSpanId": "6a0b37e31264fa20", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.308833, + "spanId": "6a0b37e31264fa20", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013459, + "spanId": "08b13b5562c6f20b", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.451083, + "spanId": "dac14f954701a7b8", + "parentSpanId": "554024b6fd76baa6", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.046625, + "spanId": "554024b6fd76baa6", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.016416, + "spanId": "b6e37f6a184c39d4", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.281291, + "spanId": "a27104b817cdfa8b", + "parentSpanId": "4172e7bfbfbda9cd", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.304041, + "spanId": "4172e7bfbfbda9cd", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014667, + "spanId": "a5e1d49f7016cc43", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.582833, + "spanId": "070eac96c83d0ed6", + "parentSpanId": "a76a7645e5f1e1a5", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.747083, + "spanId": "a76a7645e5f1e1a5", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.018334, + "spanId": "462adb15ba3c7e7e", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.888458, + "spanId": "0fbb810cd1f1b35e", + "parentSpanId": "182253a7aec6e461", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 16.914542, + "spanId": "182253a7aec6e461", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.01775, + "spanId": "2c5ea754704ebe14", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.532084, + "spanId": "2969bbddb5d7e004", + "parentSpanId": "3a0aefd37610df57", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.345333, + "spanId": "3a0aefd37610df57", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.013042, + "spanId": "49ab817d6a1f7bac", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.377333, + "spanId": "e0f55d8dac1d32f1", + "parentSpanId": "01d07244ac23cbce", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.841209, + "spanId": "01d07244ac23cbce", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020959, + "spanId": "ff24ef87601177da", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.version.verify", + "durationMs": 9.805709, + "spanId": "43451b7c272c2931", + "parentSpanId": "0a61252d03a8c331", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 11.472042, + "spanId": "0a61252d03a8c331", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + }, + { + "name": "skills.consume", + "durationMs": 268.224875, + "spanId": "1e8db9674aaf49d6", + "parentSpanId": "0cfc1dc3706e6fb1", + "attributes": { + "paperclip.task.run.run_id": "f5e68d1e0075" + } + } + ] + }, + { + "label": "warm-new-service", + "elapsedMs": 645.1124160000036, + "nativeAssetMs": 276.84833400000934, + "sqlResponseBytes": 28969, + "objectGets": 0, + "objectReadBytes": 0, + "rssBytes": 1171111936, + "cpuUserMs": 354.318, + "cpuSystemMs": 557.383, + "processPeakRssBytes": 1196851200, + "rootPoolQueries": 5, + "traceId": "196df4f3a913e668a19bec5b4e8712cc", + "phases": [ + { + "name": "skills.cache.acquire", + "durationMs": 90.46075, + "spanId": "8f0419b5e76488c0", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.rows", + "durationMs": 23.6435, + "spanId": "b6839a7d154544e3", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.030958, + "spanId": "ab99e1a18480bde3", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 20.16, + "spanId": "cbf9578fb6d65742", + "parentSpanId": "d7b003ec7bcba20c", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 22.383917, + "spanId": "d7b003ec7bcba20c", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014625, + "spanId": "372aa4b8582a91d0", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.738417, + "spanId": "edff35e5e9dcae7a", + "parentSpanId": "c6946634c6de78b0", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.657417, + "spanId": "c6946634c6de78b0", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.049083, + "spanId": "e2c1ea11ccdbb3e7", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.05575, + "spanId": "e712191554558e6c", + "parentSpanId": "621beef0cc725691", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.012375, + "spanId": "621beef0cc725691", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.020833, + "spanId": "ec045afdf6a938c2", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 10.75625, + "spanId": "3cdcc7dc7e8fd28d", + "parentSpanId": "d4ff2e88fff43840", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.678375, + "spanId": "d4ff2e88fff43840", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014458, + "spanId": "a92e593f56c85b6a", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 13.051625, + "spanId": "45aa01309fd53b1d", + "parentSpanId": "473848b8348e97a5", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 14.856584, + "spanId": "473848b8348e97a5", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017125, + "spanId": "43d6e82c6465e57f", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 12.992041, + "spanId": "2cde5272088c3d65", + "parentSpanId": "81d5bacbd7518e93", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 15.005625, + "spanId": "81d5bacbd7518e93", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.060834, + "spanId": "6364e4cf739829d0", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.14825, + "spanId": "9a9582b06d2c601b", + "parentSpanId": "69dfc28a1b8dca0a", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 12.869459, + "spanId": "69dfc28a1b8dca0a", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.015042, + "spanId": "e87803d36ea735fd", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.808791, + "spanId": "968522a5048d3ec3", + "parentSpanId": "fe4fad0f31789cb3", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.705458, + "spanId": "fe4fad0f31789cb3", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.014875, + "spanId": "75d19610b872698a", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.450458, + "spanId": "e0fc9cb907209b59", + "parentSpanId": "32e719e16f571bca", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.3125, + "spanId": "32e719e16f571bca", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.read", + "durationMs": 0.017208, + "spanId": "8f04af3139c4eaa4", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.version.verify", + "durationMs": 11.80975, + "spanId": "838d39b4b1043455", + "parentSpanId": "01a7229a9f823451", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc", + "paperclip.execution.fileCount": 1359 + } + }, + { + "name": "skills.version.materialize", + "durationMs": 13.789333, + "spanId": "01a7229a9f823451", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + }, + { + "name": "skills.consume", + "durationMs": 300.645458, + "spanId": "468d74c9b84e8a42", + "parentSpanId": "4940a84904f1a736", + "attributes": { + "paperclip.task.run.run_id": "e6eeed28e1cc" + } + } + ] + } + ] + } + ] + } +} diff --git a/doc/plans/2026-10-28-skill-package-storage.md b/doc/plans/2026-10-28-skill-package-storage.md new file mode 100644 index 00000000000..f7302c9b9ff --- /dev/null +++ b/doc/plans/2026-10-28-skill-package-storage.md @@ -0,0 +1,444 @@ +# Move installed skill packages out of PostgreSQL + +Status: implemented in this checkout, including automatic conversion, exact-commit +recovery and cache retention. Focused verification passed; the broad test run +did not complete cleanly (details below). No production changes. +Operational contract: [skill package storage](../skill-package-storage.md). +Measured results: [benchmark receipts](2026-10-28-skill-package-performance-results.json). +Investigation date: 2026-10-28. Code inspected: `1ed0af41867ba40fd1d23292d395588d18c2fcbe`. + +## Decision + +Store the exact audited installed package in the existing storage backend: S3-compatible +object storage for hosted instances, persistent local storage for local installations. +PostgreSQL retains package identities, storage receipts, versions, source tracking, +permissions and run pins. A bounded, verified disk cache supplies execution and file reads. + +New Git packages retain an immutable recipe with repository identity, exact commit, +included-file manifest and packager version. Authored and unproven legacy packages +retain independent durable bytes. Also retain the installed package's content hash: the package can exclude unsafe, oversized or unsupported files and can be +a reduced repository subset. A commit SHA alone does not identify that transformation, +provide the bytes when upstream disappears, or represent an edited local copy. + +Preserve today's packaging and access behavior during the storage migration. In particular, +do not switch all imports back to directory-only mode or infer a complete dependency +graph from Markdown references. The original dependency-loss bug was real. + +The first delivery removes file-content traffic from PostgreSQL for migrated repository +packages, avoids repeated remote reads of the same immutable package, and stops creating +packages for repositories with no installable selected skills. Deduplicating the final +per-skill runtime transport is a separate, explicitly scoped follow-up. + +## What the trace established + +### Import and refresh + +1. Project workspace creation/update calls `enqueueProjectSkillSource` inside the + workspace transaction. An eligible GitHub URL is sufficient to enqueue; this happens + before discovery. Existing legacy projects have an explicit adoption path. +2. `skillSourceSyncService.importBinding` checks authority, reserves bounded work credits, + and requests `packageMode: 'repository'`. The scanner resolves a commit and uses a + shallow, temporary Git repository to list paths and read blobs. +3. `github-skill-source.ts` discovers entrypoints, builds and audits package files, and + filters unsupported/unsafe files. Repository mode normally includes the supported + tree, with a skill-folder-plus-reference fallback for oversized repositories and a + further skill-only fallback. It does not store Git history or `.git` metadata. +4. `skillSourceService.publish` hashes `scan.repositoryFiles` and inserts their **contents** + into `company_skill_repository_snapshots.file_inventory`. Deduplication is by company + and content hash. The insert happens before the selected-valid-skill loop; a nonempty + package can therefore be stored with no discovered or selected usable skills. + `ON CONFLICT DO NOTHING` avoids a duplicate row, but still sends the proposed file + array to PostgreSQL whenever publication reaches that insert. The earlier unchanged + commit/tree shortcut avoids this path for qualifying automatic checks. +5. Each selected skill version references that snapshot and its original entrypoint. + Publication also passes `candidate.files` into `createVersion`, which persists those + contents again in `company_skill_versions.file_inventory`. Thus the full repository is + shared, but the selected skills' own files have an additional persisted copy. +6. Automatic refresh already skips downloading when the commit or root tree is unchanged. + It uses saved setup authority, leases, budgets and five-minute deadlines. An included + file change changes the package hash and versions every selected skill. This conservative + invalidation preserves access to shared and dynamically chosen dependencies. + +The similarly named fields are different: `company_skills.file_inventory` and +`company_skill_sources.repository_package.files` are file metadata; repository snapshot +and version `file_inventory` fields contain actual contents. Moving the former alone +would not address the measured payload. + +Source pointers: [project attachment](../../server/src/services/projects.ts), +[binding and identity](../../server/src/services/project-skill-source-bindings.ts), +[worker](../../server/src/services/skill-source-sync.ts), +[scanner](../../server/src/services/github-skill-source.ts), +[publication](../../server/src/services/skill-sources.ts), +[snapshot/version schema](../../packages/db/src/schema/company_skills.ts). + +### Reads and execution + +`company-skills.ts::runtimeVersionInventory` selects the entire repository snapshot, +then `repositorySkillProjection` builds a generated discovery wrapper plus the full +original tree under `.paperclip-repository/`. + +| Consumer | Current unnecessary or amplified work | Replacement | +| --- | --- | --- | +| Runtime preparation | Loads JSONB before checking the materialized directory and before entering the per-version shared promise | Resolve a small package descriptor; coalesce package loading before I/O; validate disk files against hashes | +| Skill audit | Materialization loads the repository; audit requests the inventory again | Reuse the verified package handle and manifest; audit actual delivered bytes | +| Skill detail/file tree | Loads contents, then discards them to keep paths and kinds | Read the package manifest only | +| One supporting file | Falls back to loading the whole repository inventory | Read the requested file from the verified local package cache | +| Fork, create-from-existing, disconnect-and-keep | Projects repository files; editable versions can persist a full projected tree inline | Copy from a package handle; save resulting snapshots to the same external package store | +| Cache retention | `hasRestorableSnapshot` reads repository JSONB on a process-cache miss to validate entrypoints | Verify external durability before eviction; never evict the last usable copy on storage failure | +| Version API / Studio test context | Version objects and one heartbeat join directly consume inline version inventories | Metadata-only internal queries plus explicit package hydration where content is required | + +The existing per-version disk cache does **not** prevent the initial database read. +Existing cache locks serialize publication, pinning and eviction, but do not turn +repository identity into a shared content cache. + +Each selected entrypoint also gets its own full runtime copy. Native remote launch loops +over skill bundles and stages each directory separately. Moving storage alone does not +remove this disk/transport multiplication. Never assume 79 installed skills means all +79 are assigned or transferred on every run. + +Source pointers: [runtime, audit, browsing and version creation](../../server/src/services/company-skills.ts), +[projection](../../server/src/services/skill-repository-package.ts), +[cache locking and restorability](../../server/src/services/skill-version-cache.ts), +[Studio context](../../server/src/services/heartbeat/run-preparation.ts), +[remote staging](../../server/src/services/native-runtime/remote-launch-assets.ts), +[company export](../../server/src/services/company-portability.ts). + +### Evidence and limits + +The preceding Railway investigation recorded six snapshot reads returning **93.6 MB** +in a 15-minute window: **12.94%** of observed PostgreSQL response bytes. A later metadata +inspection found a package containing **1,349 files**, **22,611,267 serialized JSON bytes**, +and **79 current referencing skills**. These are prior observations, not a fresh benchmark +of this proposal. The measured build was `71b448b9`; the later example was observed on +`b25f3e2a`. The principal import/runtime files inspected here match the earlier linked +`2e5bbde8` revision. Do not extrapolate either capture to every current instance. + +Serialized JSON bytes are neither compressed on-disk PostgreSQL size nor raw repository +size. PostgreSQL can compress/store large values out of line using +[TOAST](https://www.postgresql.org/docs/current/storage-toast.html). Selecting the complete +inventory still returns that value to the application. New snapshot payloads also add +database storage, WAL, backup and decoding work; those costs were not quantified here. + +Removing this traffic does not establish a 12.94% latency improvement, solve all Railway +slowness, or justify a specific CPU reduction. The capture measures bytes, not causality +for every slow request. + +## Why this implementation arose + +- [#14713](https://github.com/paperclipai/paperclip/pull/14713), September 30, introduced + GitHub-synchronized skills on the existing immutable-version foundation. +- [#15731](https://github.com/paperclipai/paperclip/pull/15731), October 12, fixed missing + shared scripts, policies, sibling skills and credits by preserving repository packages. + It explicitly chose shared database snapshots and self-contained per-skill delivery. +- [#16066](https://github.com/paperclipai/paperclip/pull/16066), October 13, extended that + behavior to automatic project imports and refresh. +- [#16308](https://github.com/paperclipai/paperclip/pull/16308) preserved referenced files + for oversized repositories. [#16433](https://github.com/paperclipai/paperclip/pull/16433) + subsequently made old disk copies disposable, with database restoration as the guarantee. + +The #15731 description reports strong byte/mode, offline, pinning and real runtime proof, +and explicitly acknowledges per-skill runtime duplication. Its inline reviews caught +audit/projection mismatches, lost binary/mode information, unsafe files, expensive regexes, +and per-blob Git process overhead. The review was not indifferent to correctness or cost. + +The inspected acceptance evidence does not establish a bound on PostgreSQL response bytes +for warm preparation, one-file browsing, or multiple entrypoints sharing a large package. +The supported explanation is that a legitimate dependency-preservation feature reused a +content-bearing version contract, while performance proof focused on acquisition and +functional delivery. Automatic adoption enlarged the affected workload. Original author +sessions were not identified, so this is not a claim about unrecorded reasoning. + +The preventive change is a storage boundary whose normal return type cannot contain a +repository's bytes, backed by real SQL-byte and storage-read assertions for shared packages. + +## Proposed implementation + +### 1. One package service, existing storage providers + +Add an owning service such as `skill-package-store.ts` for encoding, publication receipts, +manifest reads, verified extraction and cache acquisition. `company-skills.ts` consumes it; +the heartbeat only asks for a prepared skill. Provider details stay under `server/src/storage`. + +The existing [storage API](../../server/src/storage/types.ts) supports streamed uploads, +reads, HEAD and deletion, with [local](../../server/src/storage/local-disk-provider.ts) +and [S3](../../server/src/storage/s3-provider.ts) providers. Reuse this infrastructure. +Do not put packages in user-visible issue attachments or give them attachment retention. + +Use a versioned `tar+gzip` bundle of raw file bytes and a separate bounded JSON manifest. +The manifest contains original path, kind, byte length, SHA-256 and executable flag for +each file. Archive entries live under `files/`; reserved format metadata stays outside +that namespace. Preserve binary bytes and supported modes. No base64 repository JSON is +needed in the stored archive. Stream encoding/compression to a bounded temporary file, +hash it, then stream upload. Do not synchronously compress on the server event loop. + +Keep the existing semantic content hash for migrated packages, so migration does not +create new skill revisions. Separately record archive and manifest SHA-256 values and +format version. Do not silently change the current hashing order or rely on an S3 ETag +as a content checksum. Any future hash-algorithm change needs its own versioned identity. + +A manifest is sufficient for a file tree; a cold file read may fetch the archive once. +An archive deliberately favors simple bulk materialization over random remote file reads. +If cold single-file traffic proves material, add an indexed/range-readable format later. +Do not start with thousands of remote objects and database rows per package file. + +### 2. Metadata schema and contracts + +Extend the existing snapshot row with storage state/format and a receipt containing the +provider, server-owned archive/manifest keys, hashes, compressed/uncompressed sizes and +file count. Keep the existing snapshot ID and `(company_id, content_hash)` uniqueness. +Add company-scoped relational constraints where a new package reference is introduced. + +During migration a snapshot is one of: legacy inline; inline plus verified external copy; +external only. An explicit discriminator distinguishes absent payload from an empty +package. Keep pending upload attempts separate from externally readable versions, using +small preparation receipts/leases in the owning snapshot storage metadata. A final version +may reference only a ready package. + +Repository-backed version rows retain their snapshot and entrypoint references but stop +duplicating `candidate.files`. Preserve the directory-local view as a manifest projection. +Existing historical version inventories must be migrated from their actual stored bytes; +do not assume an older or checkpointed inventory exactly matches a current projection. +If it differs, store that inventory as its own immutable package. + +Support ordinary local, directory-only and forked versions with the same package reference +in a second compatible writer path. Keep a bounded inline legacy reader until backfill +finishes. Small searchable skill Markdown/metadata can remain on `company_skills`; the +goal is to remove file-tree payloads, not prohibit useful database text. + +Split internal `VersionMetadata` from content-bearing responses. Public +`CompanySkillVersion.fileInventory`, the version routes, Studio and export consumers +currently expect contents: preserve their response behavior through explicit hydration +from storage, or add an opt-in metadata view and move clients to it. Never return an empty +inventory to an old caller merely because the database payload was cleared. A storage +location or opaque digest is not an authorization capability and need not be exposed. + +### 3. Publication without long database transactions + +```text +authorize and scan immutable Git revision + -> audit/filter the installed package + -> obtain a bounded preparation claim for (company, content hash) + -> write manifest + archive to unique attempt-owned storage keys + -> verify stored lengths, hashes and manifest/file agreement + -> short DB transaction: recheck source revision/lease/authority; + publish ready receipt, versions, current pointers and activity together + -> settle preparation receipt and release temporary files +``` + +Perform provider I/O outside the publication transaction. Refresh's current source and +lease fencing still decides whether the result may be published. A failed upload keeps +all installed versions. A failed or uncertain commit must be reconciled before cleanup; +never delete an object simply because the transaction call returned an error. + +Use immutable unique attempt keys, with logical deduplication in PostgreSQL. This matters: +`createStorageService.putFile` currently deletes its upload key on failure. Reusing a +single deterministic key across writers could let a losing/failed writer delete a winner. +Do not use that API as a create-if-absent content-addressed store. A concurrent winner's +receipt is retained; the losing attempt's own keys become cleanup candidates only after +its outcome is known. Preserve pending receipts through restart and company deletion. + +Amazon S3's [strong read-after-write consistency](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) +helps verify a successful object write, but does not make an S3 write and SQL commit atomic. +Qualify other supported S3-compatible backends separately. Pass cancellation through the +storage facade; its current public read API omits the provider's AbortSignal capability, +and uploads need bounded cancellation/settlement too. Avoid abandoned streams after timeouts. + +### 4. Shared verified cache before per-skill projection + +Use a cache identity containing instance/storage namespace, company, format and package +content identity. Reauthorize the skill/version before every use. Deduplicating content +within a company must not expand a private project's audience; authorization stays on +the referencing skill/source, including forks and detached copies. + +Acquire a package-scoped shared promise **before** opening legacy payloads or object +storage. One cold request fetches/verifies/extracts the package; concurrent consumers reuse +it. Different service instances in one process share this work. Use filesystem locking +and atomic publication for processes sharing a volume; replicas with separate disks may +each need one fetch. Keep cache ownership distinct from authorization and version pins. + +Validate compressed size, expanded size, file count, normalized paths, duplicate/colliding +entries, per-file digest and executable mode. Reject links, devices and traversal; never +run repository code on extraction. An extracted directory or `.complete` marker alone +is not proof of byte integrity. Warm runtime validation compares actual disk files with +the hashed manifest, including unexpected files and modes, without rereading SQL payloads. +Audit continues to inspect actual content and may require new policy scans independently +of package immutability. + +Keep the current self-contained `.paperclip-repository` skill layout initially. Build it +from the shared package cache and reuse the manifest in materialization and audit. Preserve +the existing company cache lock through final publication and durable run pinning. Stage +slow downloads before that lock, then recheck authority/version identity inside it; if +the caller already holds an outer preparation lock, change that owner deliberately rather +than weakening pin safety. Cache eviction must respect active handles during staging. + +Bound cached disk bytes, active downloads, expanded temporary bytes and in-memory manifests; +do not introduce an unbounded process-wide map of decoded repository arrays. The existing +61 MiB package cap is not a bound on concurrent copies or historical cache growth. + +### 5. Avoid content work when nothing can be installed + +After listing the immutable tree, return a successful empty discovery when no valid +entrypoint path exists, before batching support blobs and constructing a repository package. +Persist the source's checked commit/tree and next check, not its contents. The initial +shallow Git fetch still transfers Git objects; do not claim zero acquisition cost. + +At publication, require at least one selected, valid candidate needing a package. A scan +with only excluded/invalid skills can retain candidate metadata and warnings without a +durable payload. A scan in which upstream skills disappear retains previously installed +versions and their packages. A later commit can still discover and import a new skill. + +Keep current unchanged-commit/tree checks and work-budget accounting. Changing the +five-minute refresh policy or adding provider events is independent work, not a prerequisite +for moving bytes out of PostgreSQL. + +## Alternatives considered + +| Alternative | Assessment | +| --- | --- | +| Store SHA; clone from upstream on use | Small database, but loses offline/deleted/revoked-upstream recovery and adds execution-time network work; does not reproduce filtered packages by itself | +| Durable bare Git mirror with retained refs | Useful for acquisition deduplication; requires durable backup, pruning/GC rules, isolation and a filtered-export contract; does not naturally cover edited/generated skills | +| Self-contained Git bundles in object storage | Viable for raw Git retention; Git distinguishes self-contained bundles from incremental bundles with prerequisites. More Git state than the installed-package contract needs | +| Compress JSONB or add a large RAM cache | Can reduce a symptom; keeps database payload storage and cold-read cost, and adds heap pressure | +| Always keep only a skill directory | Reintroduces known missing shared-file dependencies | +| Per-file content-addressed storage | Better cross-version dedup and single-file access; adds object count, manifest graph and garbage-collection complexity before that benefit is measured | +| Immutable package archive + manifest + disk cache | Recommended first implementation: existing providers, bounded bulk I/O, exact installed bytes, stable version references | + +The [Git bundle documentation](https://git-scm.com/docs/git-bundle) describes prerequisite +verification. A bundle or mirror is a real retained copy; a remote SHA is only a reference. +A reusable Git acquisition cache remains worthwhile if profiling shows repeated shallow +fetches dominate, but it must preserve the scanner's credential and caller isolation. + +## Automatic conversion, acquisition and retention + +The accepted follow-up removes the manual migration requirement. New writes default +to external packages, and a finite delivery worker converts legacy rows, independently +verifies storage and then clears duplicate payloads. It uses indexed row deadlines, +transaction notifications and restart reconstruction, rather than per-turn work or +an empty-table polling interval. The CLI is optional diagnostics. See the current +[operational contract](../skill-package-storage.md) for upgrade/rollback details. + +New Git imports persist a restore recipe alongside the package: repository identity, +source authorization reference, exact commit, precise included-file manifest and +packager version. Warm queries exclude this larger metadata. A cold miss first uses +local verified files, then stored bytes, then legacy DB bytes, then the original +Git commit using current source authorization. Every recovered file must match the +saved size/mode/hash; moving branches and current filtering rules cannot change it. +Historical packages without a provable recipe remain durable instead of inferring +an origin from mutable current skill metadata. + +Unused Git archives become eligible after 30 days, keeping current versions, explicit +pins and active/recovering runs. Metadata and recipes survive expiry. Retired storage +keys have another 30-day deletion grace. Locally authored and edited packages remain +durable while referenced. Canonical extraction retains its bounded cold-fill eviction; +version directories use existing retention; native runtime assets now have a 30-day +idle lifetime, pin-aware cleanup and a shared publication/deletion lock. A pin can +conservatively defer cleanup until the next relevant mutation/use or restart. + +Source and package writes remain company-scoped. Saved grants are rechecked on Git +acquisition, with a persisted five-minute failure backoff and existing singleflight. +No upstream network request, storage HEAD, cleanup scan, or TTL write is added to +warm package preparation. Native access timestamps are coalesced hourly on disk. + +Lifecycle ownership remains in the package/native asset services. `index.ts` only +registers them with the existing coordinator. `migration_at`/`maintenance_at` and the +existing deletion receipts are durable truth; no additional worker queue table was +introduced. Unfinished conversion blocks idle sleep. Optional retention may wait for +next ingress wake. This does not establish deployed sleep or a Cloud alarm contract. + +## Delivery slices and proof + +| Slice | Owning files/area | Exit condition | +| --- | --- | --- | +| A. Storage boundary and compatibility | New `skill-package-store.ts`; storage facade; DB/shared version descriptors; `company-skills.ts` readers | Real local and S3-compatible upload/restore round trips; old rows work; metadata reads contain no file bytes; no behavior change yet | +| B. Publisher and runtime | `skill-sources.ts`, scanner, version/fork/copy writers, package cache, cache retention | New package bytes never enter SQL; two or more selected skills share one cold package load; warm preparation does not reload it; zero-skill scan persists no package | +| C. Consumers and migration | Version routes/client contracts, Studio context, export/disconnect, bounded migration command | Old API content behavior preserved; historical pins and copies restore; full current/legacy consumer matrix passes; backfill can restart safely | +| D. Qualification and rollout | Existing skill suites and runner journey; operational docs/maps | Byte/performance evidence, fault injection, staged backfill, rollback boundary and cold recovery demonstrated | + +Slices A–D form the complete database-storage change. Keep runtime transport deduplication +separate: add a shared package bundle plus entrypoint descriptors to the runtime context, +transfer each package once, and construct each discovery entrypoint in the destination. +Preserve the existing fallback for adapters that require independent directories. Do not +use host-only symlinks or shared writable hardlinks as a shortcut; qualify read-only mounts, +copy-on-write copies or destination-local materialization against each supported transport. +This later change reduces transfer/disk multiplication; it is not needed to remove DB bytes. + +Use the existing `skill-sources`, `skill-version-cache`, `skill-version-retention`, project +skills and native runner suites. Add a representative large synthetic package and an +assembled PostgreSQL/storage benchmark, not mocks that eliminate serialization and I/O. + +| Scenario | Required evidence | +| --- | --- | +| 1,349 files / approximately 22.6 MB JSON / 1, 2 and many entrypoints | Record exact fixture size, actual PostgreSQL DataRow bytes, object GET count/bytes, CPU, peak RSS, disk bytes and preparation duration; sizes need not reproduce private content | +| One cold preparation using several skills from one package | Zero repository file-content bytes in SQL; at most one payload download per package per shared cache scope; equal delivered bytes/modes | +| Warm preparation, audit and restart with intact disk cache | Zero repository payload bytes from SQL; zero payload GETs while cache is valid; actual integrity checks retained | +| File tree and one-file read | Tree reads manifest only; first file read may fetch one archive, repeated reads reuse it; unrelated package contents never pass through SQL | +| Unchanged refresh and no-skill repository | No new package/version payload; empty scan remains discoverable on a later commit; existing installed content survives removal/failure | +| Storage failure, corrupt archive/manifest/cache, cancellation | No partially published version; old installation remains; no silent SHA refetch; unrelated requests remain responsive | +| Concurrent import, uncertain commit, process crash, deletion | No winner object deleted; persisted attempt reconciliation; stable IDs and source lease/revision fencing | +| Private project, revoked access, detached/forked content | Fresh application authorization on cache hits; no company/project boundary widening | +| Pins, cold restore, Studio, forks, disconnect, export, retention | Equivalent supported files, binaries/modes and discovery policy; historical semantics preserved without upstream | +| Backfill interrupted at each boundary | Retry changes no version IDs or contents; no early payload clearing; dual-reader rollback works | + +Report SQL byte reduction separately from object-store traffic and execution latency. +The target is **zero package file-content bytes in PostgreSQL responses for migrated +packages**, not zero database queries or zero network bytes. No percentage CPU/latency +saving is claimed until measured on the replacement. + +Update `doc/PRODUCT.md`, `doc/SPEC-implementation.md`, `doc/SPEC.md`, `doc/DATABASE.md`, +`doc/DEVELOPING.md` and recoverable-cleanup documentation to replace the database-byte +restoration guarantee with durable-package restoration. Update affected skills/version +feature and access map entries, and run `node scripts/check-maps.mjs`. Generate schema +migrations with `pnpm db:generate` and follow migration snapshot rules. Run focused checks +first; before a PR-ready implementation handoff run `pnpm -r typecheck`, `pnpm test:run` +and `pnpm build`, plus the affected real runner journey. See the operational contract +and verification record below for the checks actually completed in this checkout. + + +## Implementation verification record + +Implemented the storage migration, dual reader and guarded writer, bounded migration +CLI, deletion receipts, package/runtime caches, metadata batching and native asset +reuse. The approved follow-up adds default-on writes, automatic verified conversion, +immutable Git restore recipes, persisted acquisition backoff, automatic object cleanup +and native asset retention. The public API continues to hydrate explicit file-content responses. New source +packages are prepared before publication transactions; existing generic authoring +transactions keep their serialization boundary and use an independent upload connection. + +- Full workspace typecheck passed; the server was rebuilt after the final refinements. +- Full repository build passed. Migration numbering/safety and map contract checks passed. +- Focused package, source, company-skill, retention/cache, concurrency, heartbeat preparation + and native context suites passed. The final migration/retention rerun passed 36 tests; + the final project-source/authoring-route rerun passed 92 tests. +- Real loopback S3-compatible multipart upload and historical restoration passed; its + disposable server/data were removed after verification. +- The benchmark records 60 warm samples across 1, 2 and 10 selected entrypoints, plus + cold and fresh-service cases. It includes actual PostgreSQL wire bytes and native + asset creation. It does not prove deployed Railway latency or a live agent turn. +- The serial full-suite invocation was deliberately cancelled and replaced with the + repository's supported CI groups/shards for the same inventory. The broad run + recorded setup timeouts and assertions in the skill concurrency, native recovery, + workspace and other suites. It was stopped to isolate those failures before the + remaining chat, route and workspace groups completed. The skill concurrency case, + two representative recovery cases and the workspace assertion all passed on isolated + reruns. This is **not** a clean full-suite result or complete CI qualification. + +Per-entrypoint runtime/transport deduplication remains the separately scoped follow-up +from this design. Writers and automatic conversion now default on; the explicit false +override remains available for a coordinated reader-first rolling upgrade. + +### Automatic lifecycle follow-up + +The user approved automatic conversion, acquisition on miss and cleanup. This is +implemented without a required data-migration command. Final proof: 30 focused tests +passed, followed by 15 package/worker cases including additional missing-object and +retention regressions; 73 source/native/retention integration cases passed. Idle-safety +coverage passed, workspace typecheck/build passed, and the final server rebuild passed. +The supported fresh-cluster test path bypassed shared-template startup failures. +Two actual regressions found during development—JSONB key reordering during recipe +verification and empty native-cache startup—were fixed and covered in the passing run. + +Forty new warm benchmark samples (one and ten selected skills) plus cold/fresh-service +samples retain five ordinary SQL queries and zero object/body reads. Receipts and timing +limits are in the [operational guide](../skill-package-storage.md#automatic-lifecycle-follow-up-verification) +and the linked JSON. No production cleanup, deployment or data migration was performed. +The earlier broad full-suite limitation remains open. diff --git a/doc/recoverable-disk-cleanup.md b/doc/recoverable-disk-cleanup.md index 93f9ccb6fef..d268de28f16 100644 --- a/doc/recoverable-disk-cleanup.md +++ b/doc/recoverable-disk-cleanup.md @@ -2,9 +2,13 @@ The server materializes immutable skill versions under `/skills//__versions__//`. These are disposable -disk copies: the company-scoped version and repository snapshot rows retain the -contents, including binary files and executable flags. Normal skill resolution -rebuilds a missing copy from the database without contacting its upstream source. +disk copies: company-scoped versions reference durable packages in the configured +storage backend, or retain legacy database inventories. Normal skill resolution +rebuilds missing copies, preserving binary bytes and executable flags. New Git packages +can also restore the exact original commit from their saved recipe using current +repository access. Authored/legacy packages without that recipe require independent +external durability before eviction; unavailable storage leaves those caches intact. Shared extracted packages have a +separate cold-fill capacity policy. See [package storage](skill-package-storage.md). ## Skill version caches diff --git a/doc/skill-package-storage.md b/doc/skill-package-storage.md new file mode 100644 index 00000000000..0b2ea1fb382 --- /dev/null +++ b/doc/skill-package-storage.md @@ -0,0 +1,260 @@ +# Installed skill package storage + +Installed repository and editable-skill contents can live in the configured storage +backend instead of PostgreSQL JSONB. The database keeps company-scoped package +identities, storage receipts, source tracking, versions, permissions and run pins. +The original repository tree, wrappers, binary bytes and executable flags retain +their existing behavior. New Git packages also retain an immutable restore recipe; +locally authored/edited packages and unproven legacy packages keep a durable copy. + +## Automatic upgrade + +No data-migration command is required. The normal schema upgrade applies migrations +0398/0399. External writes and automatic conversion are enabled by default. Startup +restores pending work from indexed `migration_at` / `maintenance_at` columns. The +shared delivery coordinator converts four snapshots and four versions per batch, +reading one legacy body at a time. It verifies the hash, uploads, reads the objects +back, validates extraction, then clears the legacy JSONB. Version IDs, revision numbers +and run pins stay unchanged. Failures retain the original bytes and retry after an hour. +Bundled release inventories keep their existing small inline validation contract. + +Readers can use legacy contents while conversion is pending, including when conversion +commits between the metadata read and the content read. Conversion runs separately +from turn preparation. Completed migration history has no recurring scan. Normal +vacuum and backup retention determine when PostgreSQL reclaims physical space. + +`PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED=false` is an explicit compatibility override +for a coordinated rolling upgrade: it disables new external writes and automatic +package maintenance. All processes must support the new reader before allowing +external-only writes or clearing old payloads. Once automatic conversion runs, this +reader is the rollback floor; an older binary needs rehydrated database contents. +Use durable, backed-up local storage or S3-compatible storage for authored packages. +The extracted cache is not that storage backend. + +The optional `server/scripts/migrate-skill-packages.ts` command remains available for +diagnostics and controlled batches. It defaults to dry run, supports +`--phase=snapshots|versions|orphans|deletions`, `--company=`, `--limit=1..50`, +`--after=`, `--apply` and `--clear-legacy`. It is not part of normal installation +or upgrade. Never point it at another instance's database/storage configuration. + +## Acquisition and retention + +A missing package follows this order: verified local files, stored archive, legacy DB +contents, then an authorized exact-commit Git fetch. A new Git recipe records repository +identity, the original source, a full commit SHA, packager version, the precise included +file manifest (paths, encodings, modes, sizes and hashes), and the content hash. It is +loaded only on a cold miss, never with warm metadata queries. Recovery reads exactly +those files without rerunning discovery or today's filtering rules. A reused repository +name, wrong commit, changed file/mode/hash, missing original commit, or revoked source +credentials fails explicitly; it never substitutes the latest branch. Concurrent misses +coalesce and failed acquisition has a persisted five-minute backoff. + +After 30 days, unused Git packages with a verified recipe can lose their cached archive +and extracted copy while retaining their identity, manifest and recipe. Current library +versions, explicit agent pins, active runs and unresolved recovery keep their packages. +A source/version change schedules another eligibility check; packages held by a run or +pin are also reconsidered on restart. This is conservative retention, not an immediate +promise to purge on unassignment. Legacy packages without a provable recipe and authored +or edited copies retain their durable bytes as long as a version references them. + +Native runtime assets expire after 30 days without use. An hourly coalesced filesystem +access lease protects preparation through durable run pinning. Cleanup shares the +publication lock, preserves assets referenced by active/recovering runs, and defers if +an unfinished input cannot be interpreted safely. After a pin blocks cleanup, the next +asset use or startup reconsiders it. Missing or damaged copies rebuild from verified +source bytes. The canonical package cache retains its capacity policy below, and runtime +version directories retain their existing pin-aware retention. + +Package/version publication and deletion use transaction-aware work notifications; +local asset use signals its filesystem worker. Both reuse the delivery coordinator, +restore deadlines on startup, pause in drain/standby, and stop on shutdown. With no +pending conversion, retry, or eligible cache deadline they perform no recurring work. +The supported live writer is the owning server; independent SQL writers/other servers +need restart reconstruction or an explicit wake. No permanent DB listener is added. +Conversion is an idle-sleep blocker while unfinished. Cache/deletion times are earliest +eligibility, may wait until the next ingress wake, and do not claim an external Cloud +alarm or measured host suspension. + +## Execution and cache behavior + +- New packages store one gzip tar archive and one manifest per company/content hash. + The manifest contains paths, kinds, encodings, executable flags, byte sizes and + SHA-256 values. PostgreSQL stores keys, hashes, sizes and ownership metadata. +- Preparation uses metadata-only version/package queries. Multiple selected versions + are batched, and a shared repository descriptor is read once per preparation. +- Known selected repository skills skip unrelated library reconciliation. Selected + rows and project visibility are still read fresh. Public rows are constrained with + `access_source_id IS NULL` at the final read, so a concurrent scope change fails closed. + Aliases, legacy operational skills, missing installations and mutable local sources + retain the discovery/reconciliation path. +- A cold package download is coalesced in process; the existing company cache lock + serializes cross-process restoration and pinning. The archive and each extracted + file are verified before atomic publication. Extraction rejects unexpected entries, + path escapes, links, duplicate paths, wrong sizes, modes and digests. +- Warm checks inspect directory entries and file identity/size/mode/mtime/ctime. + Unchanged files avoid content reads and hashing. Changed identities are hashed again; + altered runtime copies are rebuilt. No marker alone establishes integrity. +- Native asset packaging also caches manifests and asset references, checking both + the selected source and delivered asset on reuse. It retains no repository bodies. + Source edits invalidate the reference; missing or corrupt copies rebuild from verified bytes. +- Concurrent cold package I/O is limited to two operations, with a queue of at most 32. + Packages are bounded to 10,000 files and 64 MiB decoded bytes. Manifest memory is + limited to 16 MiB serialized size. Repository projection metadata is shared; generated + wrappers/policies have a 16 MiB budget and 256-entry limit. The stat/digest cache holds + at most 128,000 file records, with 24 concurrent file checks. + Native skill asset manifests hold at most 32,000 file records and 128 entries. +- Shared extracted packages target 512 MiB and 64 directories per company, trimming + on cold fills only, with at most two eviction attempts and a ten-second storage + verification budget per fill. This is a soft limit during a storage outage: cleanup preserves + the last recoverable copy. Version projections retain the existing pin-aware cleanup. + Abandoned package staging directories receive a one-hour crash grace. + +Each entrypoint still has a self-contained runtime tree, and remote staging can +duplicate those trees. This change does not introduce a new runner transport format +or cross-entrypoint asset deduplication. + +## Publication, cleanup, and backups + +Source imports prepare and verify packages before opening their publication +transaction. Generic authoring calls can arrive inside an existing edit/idempotency +transaction; these retain that serialization lock during upload and use an independent +package connection to avoid a one-connection pool deadlock. No upload happens on the +unchanged warm preparation path. + +Uploads use unique attempt keys. An incomplete attempt has a durable claim with a +ten-minute expiry. Retries after expiry use fresh keys; an ambiguous database result +never triggers deletion of a possibly successful upload. A database trigger records +retired keys in `skill_package_deletions`, including company cascades. Promotion of +an upload to a published receipt does not enqueue its objects for deletion. + +The maintenance phases `orphans` and `deletions` have the same dry-run/cursor controls. +`orphans` only removes unreferenced snapshot rows older than one day whose upload +claim is settled. Foreign keys protect referenced versions. `deletions` rechecks that +no live receipt owns the keys, deletes the two objects, confirms absence, then removes +the deletion receipt. Failed attempts have at least one hour of grace; retired +published objects have 30 days. These phases also run automatically from their saved +deadlines. Do not apply a blanket bucket lifecycle rule: authored packages and legacy +packages without exact restore recipes still require durable retention. + +Database backups must be paired with object backups/history. Do not process deletion +receipts while a retained database backup outside the 30-day window still needs those +objects; extend receipt deadlines or retain object versions according to that backup +policy. A storage provider/bucket/prefix change requires copying and verifying the +objects first. The current storage facade does not find objects in an old bucket. + +## Verification + +`skill-package-store.test.ts` exercises real PostgreSQL/local storage, cold/warm +reads, same-size edits, offline historical restoration, missing durable objects, +upload recovery, company-deletion receipts, cancellation and resumable migration. +A reader that obtained its descriptor before the legacy clear pass follows the new +receipt, including when its old manifest is already cached. Existing source, +version-cache, retention, concurrency, authoring and runtime-context suites preserve +their original behavior checks. `runtime-context.test.ts` checks warm asset reuse, +source edits and repair through the normal native context builder. Additional recovery, +worker-lifecycle and asset-retention tests cover exact-commit restoration, automatic +conversion, persisted retry, idle/drain behavior and rebuilding expired native copies. + +The native preparation benchmark accepts `--package-store=external` and +`--native-assets=true`. It reports elapsed time, PostgreSQL wire response bytes, +root-pool query count, object GET count/bytes, process RSS and preparation spans. +Use `--snapshot-file=` for a representative saved package, +`--actor=agent --staging-shape=true` for authorization and unrelated-library load, +and `--network-delay-ms=5` for an emulated database delay. Synthetic local timings +are regression evidence, not a claim about deployed Railway latency. + + +To generate the same-size synthetic fixture without copying a private repository: + +```sh +python3 - <<'PYTHON' +import hashlib, json +files = [{"path": "skills/diagnostic/SKILL.md", "kind": "skill", "content": + "---\nname: diagnostic\ndescription: Native preparation benchmark\n---\nUse the bundled repository.\n"}] +for i in range(1348): + text = hashlib.shake_256(str(i).encode()).hexdigest(8300) + content = "\n".join(text[n:n + 64] for n in range(0, len(text), 64))[:16600] + files.append({"path": f"assets/{i}.txt", "kind": "reference", "content": content}) +with open("/tmp/skill-package-benchmark.json", "w") as output: + json.dump(files, output, separators=(",", ":")) +PYTHON +pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-skill-preparation.ts --snapshot-file=/tmp/skill-package-benchmark.json --package-store=external --native-assets=true --actor=agent --staging-shape=true --network-delay-ms=5 --selected-skills=10 --warm-runs=20 +``` + +The fixture generator reproduces the recorded file counts and JSON size, with new +synthetic content. Use `--selected-skills=1`, `2`, or `10` for comparable cases. + +## Measured results, 2026-10-28 + +[Recorded samples](plans/2026-10-28-skill-package-performance-results.json) use a +synthetic 1,349-file, 22,799,129-byte JSON inventory. The multi-skill cases add one +entrypoint per additional selected skill. Each case includes native asset preparation, +fresh agent authorization, 155 unrelated library skills, and 5 ms of database network +delay in each direction. The table summarizes all 20 numbered warm samples; a fresh +service instance in the same process was measured separately. + +| Selected skills | Warm median | Warm p95 | PostgreSQL response bytes | Ordinary queries | Object GETs | +| --- | ---: | ---: | ---: | ---: | ---: | +| 1 | 245 ms | 326 ms | 19,288 | 5 | 0 | +| 2 | 298 ms | 332 ms | 20,312 | 5 | 0 | +| 10 | 573 ms | 674 ms | 28,536 | 5 | 0 | + +All warm cases transferred zero package file-content bytes through SQL and zero +object bytes. Query counts exclude the dedicated advisory-lock connection; wire bytes +include the measured proxy traffic. Cold preparation fetched one manifest and one +archive, about 13.45 MB total, even with ten selected entrypoints. Cold elapsed times +were 3.7, 6.3 and 11.0 seconds respectively. These local samples ran alongside other +verification work and do not establish production latency or a CPU reduction against +an unchanged baseline. + +Median process CPU per warm preparation was 83/175/820 ms for 1/2/10 skills; filesystem +checks run concurrently, so CPU can exceed wall time. Process peak RSS was about +887/937/1,235 MB, including the application, synthetic fixture and cold allocations; +these are not incremental cache-memory measurements. Allocated instance file bytes +were about 97/152/596 MB. Runtime/native tree duplication remains visible here and is +a separate optimization. Cache budgets also mean sufficiently large selected sets or +many active companies can evict metadata/digests and require new hashing or asset +assembly; the warm timing is not a constant-time guarantee for arbitrary workloads. + +The real S3-compatible proof used a disposable loopback MinIO server built from its +official `RELEASE.2025-04-22T22-12-26Z` source. An 18,008,626-byte archive containing +22 files (18,000,095 decoded bytes) passed multipart upload, independent read-back +verification and cold restoration. `server/scripts/verify-skill-package-s3.ts` repeats +this test against a loopback endpoint and creates/deletes its own bucket and database. +No production storage or Railway instance was changed. + + +Workspace typecheck, build, migration checks and focused integration tests passed. +The attempted full repository test run did not complete cleanly: parallel shards +reported fixture setup timeouts and assertions, then were stopped for isolation. +The affected skill concurrency assertion and sampled recovery/workspace failures passed +on isolated reruns. The remaining broad inventory is not claimed as verified; see the +[implementation record](plans/2026-10-28-skill-package-storage.md#implementation-verification-record). + +## Automatic lifecycle follow-up verification + +The final implementation passed 30 focused package/recovery/worker/native tests, +then 15 package/worker tests including two additional regressions for missing stored +objects with a live receipt and retaining unproven historical bytes. Source-sync, +version-retention and native integrations passed 73 tests. The idle-safety suite also +passed. Full workspace typecheck/build passed; the final server was rebuilt after the +manifest normalization fix. Migrations and map checks passed. + +The shared PostgreSQL test-template path timed out (and exited a standalone benchmark +before measurement). The passing final runs used the supported +`PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` fresh-cluster path. Earlier retirement assertions +caught PostgreSQL JSONB key reordering; manifest serialization now preserves its v1 +wire encoding after a DB round trip. An empty native cache startup case was also fixed. + +The final 20-sample warm runs measured: + +| Selected skills | Median | p95 | SQL response bytes | Ordinary queries | Object reads | +| --- | ---: | ---: | ---: | ---: | ---: | +| 1 | 256 ms | 302 ms | 19,423 | 5 | 0 | +| 10 | 691 ms | 816 ms | 28,969 | 5 | 0 | + +All warm samples still transfer zero package bodies through SQL and zero object bytes. +The larger restore recipe stays out of those queries. Ten-skill cold preparation was +slower on this busy host; the raw receipts retain that result as well as warm samples. +The earlier full repository test attempt remains incomplete; this follow-up does not +claim a clean full-suite result, deployed Railway measurements, or a live agent run. diff --git a/maps/agent-access-map/skills.md b/maps/agent-access-map/skills.md index 71776058621..55ce78c0295 100644 --- a/maps/agent-access-map/skills.md +++ b/maps/agent-access-map/skills.md @@ -13,6 +13,22 @@ Each capability needs operation-level review before its entire scope can be qualified. No live agent execution is claimed. Follow the [map contract](../README.md) for format, permission checks, evidence and maintenance. +Cold Git package recovery rechecks the original source's current setup identity, +company membership, policy and repository grant. It never borrows another company's +credentials or substitutes a moving branch for the saved commit. A missing/revoked +source fails with a reconnect error and persisted retry backoff. Warm access still +uses the ordinary skill/project permission check. + +External package storage changes persistence, not agent authority. Runtime selection +still checks current company/project access before consulting immutable package caches; +no storage key is exposed as an agent capability. Public selected rows are constrained +to `access_source_id IS NULL` at their final read. Warm package and native asset caches +hold file metadata, never authorization decisions. The +[concurrency regression](../../server/src/__tests__/company-skills-runtime-concurrency.test.ts) +covers separate actors and revoked project access; the +[package regression](../../server/src/__tests__/skill-package-store.test.ts) covers +company isolation and warm-cache behavior. This does not claim live agent qualification. + ## `discovery` Capability ID: `skills/discovery`. diff --git a/maps/feature-map/skills.md b/maps/feature-map/skills.md index b3db94c6cc9..d98ad2d38e3 100644 --- a/maps/feature-map/skills.md +++ b/maps/feature-map/skills.md @@ -81,6 +81,14 @@ version pins. The [concurrency regression](../../server/src/__tests__/company-sk checks shared work and separate actor access; the cache tests check that cleanup cannot remove a snapshot before the run stores its pins. +The [package storage regression](../../server/src/__tests__/skill-package-store.test.ts) +covers external snapshots, automatic legacy conversion and exact-commit recovery. A warm selected-repository +check must perform no object reads or repository-content SQL. Source/library +reconciliation remains available for discovery and mutable sources. The native context +regression checks that warm asset packaging reuses verified files, repairs missing/corrupt copies, +and notices source edits. Verify both writer-switch states during rollout; see +[package storage](../../doc/skill-package-storage.md). + ## Gotchas - The release picker requires `enableBetaSkills`. Do not invent an Instructions diff --git a/packages/db/src/migrations/0401_icy_sister_grimm.sql b/packages/db/src/migrations/0401_icy_sister_grimm.sql new file mode 100644 index 00000000000..cf9ca8d7077 --- /dev/null +++ b/packages/db/src/migrations/0401_icy_sister_grimm.sql @@ -0,0 +1,81 @@ +CREATE TABLE IF NOT EXISTS "skill_package_deletions" ( + "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL, + "company_id" uuid NOT NULL, + "provider" text NOT NULL, + "archive_key" text NOT NULL, + "manifest_key" text NOT NULL, + "delete_after" timestamp with time zone NOT NULL +); +--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ALTER COLUMN "file_inventory" SET DEFAULT '[]'::jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "storage" jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "upload" jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "restore" jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "migration_at" timestamp with time zone DEFAULT now();--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "maintenance_at" timestamp with time zone DEFAULT now();--> statement-breakpoint +ALTER TABLE "company_skill_repository_snapshots" ADD COLUMN IF NOT EXISTS "restore_retry_at" timestamp with time zone;--> statement-breakpoint +ALTER TABLE "company_skill_versions" ADD COLUMN IF NOT EXISTS "package_snapshot_id" uuid;--> statement-breakpoint +ALTER TABLE "company_skill_versions" ADD COLUMN IF NOT EXISTS "package_file_map" jsonb;--> statement-breakpoint +ALTER TABLE "company_skill_versions" ADD COLUMN IF NOT EXISTS "migration_at" timestamp with time zone DEFAULT now();--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "skill_package_deletions_due_idx" ON "skill_package_deletions" USING btree ("delete_after","id");--> statement-breakpoint +DO $$ BEGIN +ALTER TABLE "company_skill_versions" ADD CONSTRAINT "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk" FOREIGN KEY ("package_snapshot_id") REFERENCES "public"."company_skill_repository_snapshots"("id") ON DELETE restrict ON UPDATE no action; +EXCEPTION WHEN duplicate_object THEN NULL; +END $$;--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_snapshots_migration_idx" ON "company_skill_repository_snapshots" USING btree ("migration_at") WHERE "company_skill_repository_snapshots"."migration_at" is not null;--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_snapshots_maintenance_idx" ON "company_skill_repository_snapshots" USING btree ("maintenance_at") WHERE "company_skill_repository_snapshots"."maintenance_at" is not null;--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_versions_migration_idx" ON "company_skill_versions" USING btree ("migration_at") WHERE "company_skill_versions"."migration_at" is not null;--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_versions_package_snapshot_idx" ON "company_skill_versions" USING btree ("package_snapshot_id");--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "company_skill_versions_repository_snapshot_idx" ON "company_skill_versions" USING btree ("repository_snapshot_id"); +--> statement-breakpoint +CREATE OR REPLACE FUNCTION paperclip_queue_skill_package_deletion() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF OLD.storage IS NOT NULL AND (TG_OP = 'DELETE' OR OLD.storage IS DISTINCT FROM NEW.storage) THEN + INSERT INTO skill_package_deletions(company_id, provider, archive_key, manifest_key, delete_after) + VALUES (OLD.company_id, OLD.storage->>'provider', OLD.storage->>'archiveKey', OLD.storage->>'manifestKey', now() + interval '30 days'); + END IF; + IF OLD.upload IS NOT NULL AND (TG_OP = 'DELETE' OR ( + OLD.upload->>'token' IS DISTINCT FROM NEW.upload->>'token' + AND OLD.upload->>'archiveKey' IS DISTINCT FROM NEW.storage->>'archiveKey' + )) THEN + INSERT INTO skill_package_deletions(company_id, provider, archive_key, manifest_key, delete_after) + VALUES (OLD.company_id, OLD.upload->>'provider', OLD.upload->>'archiveKey', OLD.upload->>'manifestKey', + greatest(now() + interval '1 hour', (OLD.upload->>'expiresAt')::timestamptz)); + END IF; + RETURN NULL; +END; +$$; +--> statement-breakpoint +DROP TRIGGER IF EXISTS skill_package_deletion_receipts ON company_skill_repository_snapshots; +--> statement-breakpoint +CREATE TRIGGER skill_package_deletion_receipts AFTER UPDATE OF storage, upload OR DELETE +ON company_skill_repository_snapshots FOR EACH ROW EXECUTE FUNCTION paperclip_queue_skill_package_deletion(); + +--> statement-breakpoint +CREATE OR REPLACE FUNCTION paperclip_schedule_skill_package_cleanup() RETURNS trigger LANGUAGE plpgsql AS $$ +DECLARE version_id uuid; +BEGIN + IF TG_TABLE_NAME = 'company_skills' THEN + version_id := OLD.current_version_id; + IF version_id IS NOT NULL AND (TG_OP = 'DELETE' OR version_id IS DISTINCT FROM NEW.current_version_id) THEN + UPDATE company_skill_repository_snapshots SET maintenance_at = now() + interval '30 days' + WHERE id IN (SELECT repository_snapshot_id FROM company_skill_versions WHERE id = version_id + UNION SELECT package_snapshot_id FROM company_skill_versions WHERE id = version_id); + END IF; + ELSE + UPDATE company_skill_repository_snapshots SET maintenance_at = now() + interval '1 day' + WHERE id = OLD.repository_snapshot_id OR id = OLD.package_snapshot_id; + END IF; + RETURN NULL; +END; +$$; +--> statement-breakpoint +DROP TRIGGER IF EXISTS skill_package_current_cleanup ON company_skills; +--> statement-breakpoint +CREATE TRIGGER skill_package_current_cleanup AFTER UPDATE OF current_version_id OR DELETE ON company_skills +FOR EACH ROW EXECUTE FUNCTION paperclip_schedule_skill_package_cleanup(); +--> statement-breakpoint +DROP TRIGGER IF EXISTS skill_package_version_cleanup ON company_skill_versions; +--> statement-breakpoint +CREATE TRIGGER skill_package_version_cleanup AFTER DELETE ON company_skill_versions +FOR EACH ROW EXECUTE FUNCTION paperclip_schedule_skill_package_cleanup(); diff --git a/packages/db/src/schema/company_skills.ts b/packages/db/src/schema/company_skills.ts index d6ce60b2aeb..ed573b983e2 100644 --- a/packages/db/src/schema/company_skills.ts +++ b/packages/db/src/schema/company_skills.ts @@ -70,14 +70,77 @@ export type CompanySkillVersionFileInventoryEntry = CompanySkillFileInventoryEnt executable?: boolean; }; +export type SkillPackageStorageReceipt = { + format: "tar-gzip-v1"; + provider: "local_disk" | "s3"; + archiveKey: string; + archiveSha256: string; + archiveBytes: number; + manifestKey: string; + manifestSha256: string; + manifestBytes: number; + fileCount: number; + totalBytes: number; +}; + +export type SkillPackageUpload = { + token: string; + legacy?: boolean; + provider: "local_disk" | "s3"; + archiveKey: string; + manifestKey: string; + expiresAt: string; +}; + +/** Exact input to the v1 packager. File bodies never live in this recipe. */ +export type SkillPackageRestoreRecipe = { + version: 1; + sourceId: string; + repositoryId: string; + repositoryUrl: string; + commitSha: string; + skillPath: string | null; + manifest: { + format: "paperclip-skill-package/v1"; + contentHash: string; + totalBytes: number; + files: Array<{ path: string; kind: CompanySkillFileInventoryEntry["kind"]; encoding: "utf8" | "base64"; + executable: boolean; size: number; sha256: string }>; + }; +}; + +export type SkillPackageFileMapping = CompanySkillFileInventoryEntry & { sourcePath: string }; + +/** Durable deletion receipts survive company/snapshot deletion until storage confirms removal. */ +export const skillPackageDeletions = pgTable("skill_package_deletions", { + id: uuid("id").primaryKey().defaultRandom(), + companyId: uuid("company_id").notNull(), + provider: text("provider").notNull(), + archiveKey: text("archive_key").notNull(), + manifestKey: text("manifest_key").notNull(), + deleteAfter: timestamp("delete_after", { withTimezone: true }).notNull(), +}, table => ({ due: index("skill_package_deletions_due_idx").on(table.deleteAfter, table.id) })); + /** Immutable repository bytes shared by all skill versions from the same package. */ export const companySkillRepositorySnapshots = pgTable("company_skill_repository_snapshots", { id: uuid("id").primaryKey().defaultRandom(), companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }), contentHash: text("content_hash").notNull(), - fileInventory: jsonb("file_inventory").$type().notNull(), + // Legacy payload only. New packages retain bytes in the configured storage backend. + fileInventory: jsonb("file_inventory").$type().notNull().default([]), + storage: jsonb("storage").$type(), + upload: jsonb("upload").$type(), + restore: jsonb("restore").$type(), + // These deadlines are consumed by the delivery coordinator, never polled per turn. + migrationAt: timestamp("migration_at", { withTimezone: true }).defaultNow(), + maintenanceAt: timestamp("maintenance_at", { withTimezone: true }).defaultNow(), + restoreRetryAt: timestamp("restore_retry_at", { withTimezone: true }), createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), -}, table => ({ content: uniqueIndex("company_skill_repository_snapshots_content_idx").on(table.companyId, table.contentHash) })); +}, table => ({ + content: uniqueIndex("company_skill_repository_snapshots_content_idx").on(table.companyId, table.contentHash), + migration: index("company_skill_snapshots_migration_idx").on(table.migrationAt).where(sql`${table.migrationAt} is not null`), + maintenance: index("company_skill_snapshots_maintenance_idx").on(table.maintenanceAt).where(sql`${table.maintenanceAt} is not null`), +})); export const companySkillVersions = pgTable( "company_skill_versions", @@ -87,6 +150,9 @@ export const companySkillVersions = pgTable( companySkillId: uuid("company_skill_id").notNull().references(() => companySkills.id, { onDelete: "cascade" }), repositorySnapshotId: uuid("repository_snapshot_id").references(() => companySkillRepositorySnapshots.id, { onDelete: "restrict" }), repositorySkillPath: text("repository_skill_path"), + packageSnapshotId: uuid("package_snapshot_id").references(() => companySkillRepositorySnapshots.id, { onDelete: "restrict" }), + packageFileMap: jsonb("package_file_map").$type(), + migrationAt: timestamp("migration_at", { withTimezone: true }).defaultNow(), revisionNumber: integer("revision_number").notNull(), label: text("label"), releaseId: text("release_id"), @@ -98,6 +164,9 @@ export const companySkillVersions = pgTable( createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), }, (table) => ({ + migrationIdx: index("company_skill_versions_migration_idx").on(table.migrationAt).where(sql`${table.migrationAt} is not null`), + packageSnapshotIdx: index("company_skill_versions_package_snapshot_idx").on(table.packageSnapshotId), + repositorySnapshotIdx: index("company_skill_versions_repository_snapshot_idx").on(table.repositorySnapshotId), companySkillRevisionUniqueIdx: uniqueIndex("company_skill_versions_skill_revision_idx").on( table.companySkillId, table.revisionNumber, diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index 94677eb633d..a8b1e946615 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -193,13 +193,14 @@ export { } from "./tool_access.js"; export { companySkills, - companySkillVersions, companySkillRepositorySnapshots, + companySkillVersions, companySkillRepositorySnapshots, skillPackageDeletions, companySkillStars, companySkillComments, companySkillTestInputs, companySkillTestRunTemplates, companySkillTestRuns, } from "./company_skills.js"; +export type { SkillPackageStorageReceipt, SkillPackageUpload, SkillPackageRestoreRecipe, SkillPackageFileMapping } from "./company_skills.js"; export { plugins } from "./plugins.js"; export { pluginConfig } from "./plugin_config.js"; export { pluginCompanySettings } from "./plugin_company_settings.js"; diff --git a/server/package.json b/server/package.json index 46f9beff067..0265e98d04e 100644 --- a/server/package.json +++ b/server/package.json @@ -115,6 +115,7 @@ "smol-toml": "^1.9.1", "ssh2": "^1.17.0", "svix": "1.76.1", + "tar-stream": "3.2.0", "ws": "^8.22.0", "zod": "^4.4.3", "@opentelemetry/resources": "2.10.0" @@ -142,6 +143,7 @@ "@types/node": "^24.0.0", "@types/sharp": "^0.32.0", "@types/supertest": "^7.2.1", + "@types/tar-stream": "^3.1.5", "@types/ws": "^8.18.2", "cross-env": "^10.1.0", "supertest": "^7.0.0", diff --git a/server/scripts/benchmark-native-skill-preparation.ts b/server/scripts/benchmark-native-skill-preparation.ts index da1310afdb0..c4897afaf3e 100644 --- a/server/scripts/benchmark-native-skill-preparation.ts +++ b/server/scripts/benchmark-native-skill-preparation.ts @@ -18,11 +18,20 @@ import { authUsers, companyMemberships, companySkillSources, heartbeatRuns, proj import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; import { createRunPreparationTiming, withRunExecutionTiming } from "../src/services/execution-preparation-timing.js"; import { repositorySkillProjection } from "../src/services/skill-repository-package.js"; +import { materializeSelectedSkills } from "../src/services/native-runtime/runtime-context.js"; +import { skillPackageStore } from "../src/services/skill-package-store.js"; +import { getStorageService } from "../src/storage/index.js"; import { skillFileBytes, skillSnapshotHash } from "../src/services/skill-snapshot.js"; const argument = (name: string) => process.argv.find(value => value.startsWith(`${name}=`))?.slice(name.length + 1); const exportOtlp = argument("--export-otlp") === "true"; const authenticated = argument("--actor") === "agent"; +const warmRuns = Number(argument("--warm-runs") ?? 2); +assert.ok(Number.isSafeInteger(warmRuns) && warmRuns >= 1 && warmRuns <= 100); +const selectedSkillCount = Number(argument("--selected-skills") ?? 1); +assert.ok(Number.isSafeInteger(selectedSkillCount) && selectedSkillCount >= 1 && selectedSkillCount <= 100); +const nativeAssets = argument("--native-assets") === "true"; +const externalPackages = argument("--package-store") === "external"; const stagingShape = argument("--staging-shape") === "true"; // This standalone benchmark owns its tracer. Load the service graph with the // app-wide SDK disabled, then restore the operator's export configuration. @@ -44,7 +53,9 @@ const sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { }).trim(); const sourceHash = createHash("sha256"); for (const file of [import.meta.url, new URL("../src/services/company-skills.ts", import.meta.url), - new URL("../src/services/skill-version-cache.ts", import.meta.url), new URL("../src/services/execution-preparation-timing.ts", import.meta.url)]) { + new URL("../src/services/skill-version-cache.ts", import.meta.url), new URL("../src/services/execution-preparation-timing.ts", import.meta.url), + ...["skill-package-store", "skill-package-runtime", "skill-package-files", "skill-package-cache", "file-manifest-cache", + "skill-package-recovery", "skill-package-work", "native-runtime/runtime-context", "native-runtime/skill-asset-cache", "native-runtime/runtime-asset-retention"].map(name => new URL(`../src/services/${name}.ts`, import.meta.url))]) { sourceHash.update(await fs.readFile(new URL(file))); } const sourceChangesHash = sourceHash.digest("hex"); @@ -55,6 +66,14 @@ const files: CompanySkillVersionFileInventoryEntry[] = snapshotFile ? JSON.parse(await fs.readFile(snapshotFile, "utf8")) : [{ path: entrypoint, kind: "skill", content: "---\nname: diagnostic\ndescription: Native preparation benchmark\n---\nReport the model ID.\n" }, ...Array.from({ length: 207 }, (_, index) => ({ path: `assets/${index}.txt`, kind: "reference" as const, content: "x".repeat(52_000) }))]; +const entrypoints = [entrypoint]; +for (let i = 1; i < selectedSkillCount; i++) { + const source = files.find(file => file.path === entrypoint)!; + const next = `skills/diagnostic-${i}/SKILL.md`; + assert.ok(!files.some(file => file.path === next)); + files.push({ ...source, path: next, content: source.content.replace("name: diagnostic", `name: diagnostic-${i}`) }); + entrypoints.push(next); +} const projected = repositorySkillProjection(files, entrypoint); const previous = { home: process.env.PAPERCLIP_HOME, instance: process.env.PAPERCLIP_INSTANCE_ID, endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }; const home = await fs.mkdtemp(path.join(os.tmpdir(), "native-skill-benchmark-")); @@ -113,12 +132,15 @@ try { catch (error) { await cleanup(); throw error; } return { connectionString, cleanup }; } + console.error("benchmark: starting isolated database"); database = await startDatabase(); + console.error("benchmark: seeding isolated fixture"); const networkDelayMs = Number(argument("--network-delay-ms") ?? 0); assert.ok(Number.isFinite(networkDelayMs) && networkDelayMs >= 0 && networkDelayMs <= 100); const target = new URL(database.connectionString); const upstreamPort = Number(target.port); - proxy = networkDelayMs ? createServer(client => { + let sqlResponseBytes = 0; + proxy = createServer(client => { const upstream = connect({ host: "127.0.0.1", port: upstreamPort }); for (const socket of [client, upstream]) { socket.setNoDelay(true); @@ -126,6 +148,7 @@ try { socket.on("error", () => { client.destroy(); upstream.destroy(); }); socket.on("close", () => { connections.delete(socket); }); } + upstream.on("data", chunk => { sqlResponseBytes += chunk.length; }); const relay = (source: Socket, destination: Socket) => { let pendingBytes = 0; source.on("data", chunk => { @@ -142,7 +165,7 @@ try { source.on("end", () => setTimeout(() => destination.end(), networkDelayMs)); }; relay(client, upstream); relay(upstream, client); - }) : null; + }); if (proxy) { const server = proxy; await new Promise((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); }); @@ -154,8 +177,7 @@ try { const companyId = randomUUID(); const agentId = randomUUID(); const userId = randomUUID(); - const skillId = randomUUID(); - const key = `company/${companyId}/diagnostic`; + const selections = new Map(); const samples: unknown[] = []; let rootPoolQueries = 0; db.$client.options.debug = () => { rootPoolQueries++; }; @@ -188,27 +210,50 @@ try { }))); await db.insert(companySkills).values({ companyId, key: "fixture/catalog", slug: "catalog", name: "Unselected catalog", sourceType: "catalog", markdown: "x".repeat(1_578) }); } - await db.insert(companySkills).values({ id: skillId, companyId, key, slug: "diagnostic", name: "Diagnostic", - markdown: files.find(file => file.path === entrypoint)!.content, sourceType: "github", - sourceLocator: "https://example.invalid/immutable-fixture", metadata: { packageMode: "repository", snapshotHash: "fixture" } }); - const [snapshot] = await db.insert(companySkillRepositorySnapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); - const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, revisionNumber: 1, - repositorySnapshotId: snapshot!.id, repositorySkillPath: entrypoint }).returning(); - await db.update(companySkills).set({ currentVersionId: version!.id }).where(eq(companySkills.id, skillId)); + const snapshotId = externalPackages ? await skillPackageStore(db).prepare(companyId, files) + : (await db.insert(companySkillRepositorySnapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning())[0]!.id; + let objectGets = 0, objectReadBytes = 0; + const backend = getStorageService(); + const getObject = backend.getObject.bind(backend); + backend.getObject = async (...args) => { + const object = await getObject(...args); + objectGets++; objectReadBytes += object.contentLength ?? 0; + return object; + }; + for (const [index, selectedPath] of entrypoints.entries()) { + const skillId = randomUUID(); + const key = `company/${companyId}/diagnostic-${index}`; + await db.insert(companySkills).values({ id: skillId, companyId, key, slug: `diagnostic-${index}`, name: `Diagnostic ${index}`, + markdown: files.find(file => file.path === selectedPath)!.content, sourceType: "github", + sourceLocator: "https://example.invalid/immutable-fixture", metadata: { packageMode: "repository", snapshotHash: "fixture" } }); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, revisionNumber: 1, + repositorySnapshotId: snapshotId, repositorySkillPath: selectedPath }).returning(); + await db.update(companySkills).set({ currentVersionId: version!.id }).where(eq(companySkills.id, skillId)); + selections.set(key, version!.id); + } const benchmarkDb = db; const service = companySkillService(db); - for (const label of ["cold", "warm-1", "warm-2", "warm-new-service"]) { + console.error("benchmark: measuring preparations"); + for (const label of ["cold", ...Array.from({ length: warmRuns }, (_, index) => `warm-${index + 1}`), "warm-new-service"]) { exporter.reset(); rootPoolQueries = 0; const selected: ReturnType = label === "warm-new-service" ? companySkillService(db) : service; const runId = randomUUID(); if (authenticated) await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", responsibleUserId: userId, contextSnapshot: { executionPolicy: {} } }); rootPoolQueries = 0; + sqlResponseBytes = 0; objectGets = 0; objectReadBytes = 0; + let nativeAssetMs = 0; + const cpuStarted = process.cpuUsage(); const measuredStarted = performance.now(); const resolve = () => selected.withRuntimeSkillEntries(companyId, { - onlyKeys: new Set([key]), versionSelections: new Map([[key, version!.id]]), snapshotLatest: true, + onlyKeys: new Set(selections.keys()), versionSelections: selections, snapshotLatest: true, }, async entries => { - if (authenticated) await benchmarkDb.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ skillVersions: { [key]: version!.id }, skillSelectionSnapshot: true })}::jsonb` }).where(eq(heartbeatRuns.id, runId)); + if (authenticated) await benchmarkDb.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ skillVersions: Object.fromEntries(selections), skillSelectionSnapshot: true })}::jsonb` }).where(eq(heartbeatRuns.id, runId)); + if (nativeAssets) { + const started = performance.now(); + await materializeSelectedSkills({ paperclipSkillSync: { desiredSkills: [...selections.keys()] } }, entries, true); + nativeAssetMs = performance.now() - started; + } return entries; }); const entries: Awaited> = await withRunExecutionTiming(runId, () => createRunPreparationTiming(runId)("configuration_skills", () => authenticated @@ -216,11 +261,15 @@ try { : resolve())); const elapsedMs = performance.now() - measuredStarted; const queryCount = rootPoolQueries; - assert.equal(entries.length, 1); + const cpu = process.cpuUsage(cpuStarted); + const measuredIo = { nativeAssetMs, sqlResponseBytes, objectGets, objectReadBytes, rssBytes: process.memoryUsage().rss, + cpuUserMs: cpu.user / 1000, cpuSystemMs: cpu.system / 1000, processPeakRssBytes: process.resourceUsage().maxRSS * 1024 }; + assert.equal(entries.length, selectedSkillCount); + for (const entry of entries) assert.equal(entry.sourceStatus, "available", entry.missingDetail ?? "Skill unavailable"); assert.equal(entries[0]!.sourceStatus, "available", entries[0]!.missingDetail ?? "Skill unavailable"); // Verify the whole delivered repository, not just its authored SKILL.md. for (const file of projected) { - const location = path.join(entries[0]!.source, file.path); + const location = path.join(entries.find(entry => entry.key === selections.keys().next().value)!.source, file.path); assert.deepEqual(await fs.readFile(location), skillFileBytes(file)); if (process.platform !== "win32") assert.equal(Boolean((await fs.stat(location)).mode & 0o111), Boolean(file.executable)); } @@ -230,17 +279,32 @@ try { assert.ok(spans.some(span => span.name === "skills.version.verify" && span.attributes["paperclip.execution.fileCount"] === projected.length)); assert.equal(new Set(spans.map(span => span.spanContext().traceId)).size, 1); assert.ok(spans.every(span => Object.keys(span.attributes).every(key => ["paperclip.task.run.run_id", "paperclip.execution.fileCount", "paperclip.execution.bytes"].includes(key)))); - samples.push({ label, elapsedMs, rootPoolQueries: queryCount, traceId: spans[0]!.spanContext().traceId, + samples.push({ label, elapsedMs, ...measuredIo, rootPoolQueries: queryCount, traceId: spans[0]!.spanContext().traceId, phases: spans.filter(span => span.name.startsWith("skills.")).map(span => ({ name: span.name, durationMs: span.duration[0] * 1000 + span.duration[1] / 1e6, spanId: span.spanContext().spanId, parentSpanId: span.parentSpanContext?.spanId, attributes: span.attributes })) }); - console.error(JSON.stringify({ label, elapsedMs, rootPoolQueries: queryCount })); + console.error(JSON.stringify({ label, elapsedMs, ...measuredIo, rootPoolQueries: queryCount })); } + async function diskUsage(root: string): Promise<{ allocatedBytes: number; fileBytes: number; files: number }> { + const total = { allocatedBytes: 0, fileBytes: 0, files: 0 }; + for (const item of await fs.readdir(root, { withFileTypes: true })) { + const target = path.join(root, item.name); + if (item.isDirectory()) { + const child = await diskUsage(target); + total.allocatedBytes += child.allocatedBytes; total.fileBytes += child.fileBytes; total.files += child.files; + } else if (item.isFile()) { + const stat = await fs.stat(target); + total.allocatedBytes += stat.blocks * 512; total.fileBytes += stat.size; total.files++; + } + } + return total; + } + const instanceDisk = await diskUsage(home); console.log(JSON.stringify({ schema: "paperclip.native-skill-preparation-benchmark.v1", sourceRevision, sourceChangesHash, exportedOtlp: exportOtlp, source: snapshotFile ? "supplied immutable snapshot" : "synthetic repository", - networkDelayMsPerDirection: networkDelayMs, authenticated, stagingShape, + networkDelayMsPerDirection: networkDelayMs, authenticated, stagingShape, externalPackages, nativeAssets, selectedSkillCount, instanceDisk, repositoryHash: skillSnapshotHash(files), repositoryFiles: files.length, runtimeFiles: projected.length, runtimeBytes: projected.reduce((sum, file) => sum + skillFileBytes(file).length, 0), samples }, null, 2)); } finally { @@ -256,5 +320,12 @@ try { for (const [name, value] of Object.entries({ PAPERCLIP_HOME: previous.home, PAPERCLIP_INSTANCE_ID: previous.instance, OTEL_EXPORTER_OTLP_ENDPOINT: previous.endpoint })) { if (value === undefined) delete process.env[name]; else process.env[name] = value; } + async function writableDirectories(root: string): Promise { + const stat = await fs.lstat(root); + if (!stat.isDirectory() || stat.isSymbolicLink()) return; + await fs.chmod(root, 0o700); + for (const child of await fs.readdir(root, { withFileTypes: true })) if (child.isDirectory()) await writableDirectories(path.join(root, child.name)); + } + await writableDirectories(home); await fs.rm(home, { recursive: true, force: true }); } diff --git a/server/scripts/migrate-skill-packages.ts b/server/scripts/migrate-skill-packages.ts new file mode 100644 index 00000000000..1f089f430c4 --- /dev/null +++ b/server/scripts/migrate-skill-packages.ts @@ -0,0 +1,21 @@ +/** Run from the server package with its configured storage and DATABASE_URL. */ +import { createDb } from "@paperclipai/db"; +import { skillPackageMaintenance } from "../src/services/skill-package-maintenance.js"; + +const value = (flag: string) => process.argv.find(arg => arg.startsWith(`${flag}=`))?.slice(flag.length + 1); +const phase = value("--phase") ?? "snapshots"; +const url = process.env.DATABASE_URL; +if (!url) throw new Error("DATABASE_URL must explicitly select the database to migrate"); +const limit = Number(value("--limit") ?? 10); +if (!Number.isSafeInteger(limit) || limit < 1 || limit > 50) throw new Error("--limit must be between 1 and 50"); +const db = createDb(url, { maxConnections: 2 }); +try { + const service = skillPackageMaintenance(db); + const actions = { snapshots: service.migrateSnapshots, versions: service.migrateVersions, + orphans: service.collectOrphans, deletions: service.deleteObjects }; + const action = actions[phase as keyof typeof actions]; + if (!action) throw new Error("--phase must be snapshots, versions, orphans, or deletions"); + const apply = process.argv.includes("--apply"); + const result = await action({ apply, clearLegacy: process.argv.includes("--clear-legacy"), limit, after: value("--after"), companyId: value("--company") }); + console.log(JSON.stringify({ phase, apply, ...result })); +} finally { await db.$client.end({ timeout: 1 }); } diff --git a/server/scripts/verify-skill-package-s3.ts b/server/scripts/verify-skill-package-s3.ts new file mode 100644 index 00000000000..aaefcc4186b --- /dev/null +++ b/server/scripts/verify-skill-package-s3.ts @@ -0,0 +1,59 @@ +/** Disposable, loopback-only S3 compatibility check. No application data is used. */ +import assert from "node:assert/strict"; +import { randomBytes, randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { S3Client, CreateBucketCommand, DeleteBucketCommand } from "@aws-sdk/client-s3"; +import { companies, createDb, startEmbeddedPostgresTestDatabase } from "@paperclipai/db"; +import { createStorageService } from "../src/storage/service.js"; +import { createS3StorageProvider } from "../src/storage/s3-provider.js"; +import { skillPackagePaths, skillPackageStore } from "../src/services/skill-package-store.js"; +import { skillSnapshotHash } from "../src/services/skill-snapshot.js"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; + +const endpoint = process.env.PAPERCLIP_SKILL_PACKAGE_TEST_S3_ENDPOINT; +assert.ok(endpoint, "Set PAPERCLIP_SKILL_PACKAGE_TEST_S3_ENDPOINT to a disposable loopback S3 server"); +assert.ok(["localhost", "127.0.0.1", "[::1]"].includes(new URL(endpoint).hostname), "Only loopback test storage is allowed"); +const region = "us-east-1", bucket = `skill-package-test-${randomUUID()}`; +const client = new S3Client({ region, endpoint, forcePathStyle: true }); +const database = await startEmbeddedPostgresTestDatabase("skill-package-s3-"); +const db = createDb(database.connectionString); +const home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-package-s3-")); +const previousHome = process.env.PAPERCLIP_HOME; +process.env.PAPERCLIP_HOME = home; +const storage = createStorageService(createS3StorageProvider({ bucket, region, endpoint, forcePathStyle: true })); +const companyId = randomUUID(); +const keys: string[] = []; +let bucketCreated = false; +try { + await client.send(new CreateBucketCommand({ Bucket: bucket })); + bucketCreated = true; + await db.insert(companies).values({ id: companyId, name: "Disposable S3 check", issuePrefix: "S3T" }); + const files: CompanySkillVersionFileInventoryEntry[] = [ + { path: "SKILL.md", kind: "skill", content: "---\nname: s3-check\ndescription: Disposable compatibility check\n---\nTest only.\n" }, + ...Array.from({ length: 20 }, (_, index) => ({ path: `assets/${index}.bin`, kind: "asset" as const, + content: randomBytes(900_000).toString("base64"), encoding: "base64" as const })), + { path: "scripts/check.sh", kind: "script", content: "#!/bin/sh\nexit 0\n", executable: true }, + ]; + const store = skillPackageStore(db, storage); + const id = await store.prepare(companyId, files); + const receipt = (await store.descriptor(companyId, id)).storage!; + keys.push(receipt.archiveKey, receipt.manifestKey); + assert.ok(receipt.archiveBytes > 16 * 1024 * 1024, "Exercise at least three multipart upload parts"); + assert.equal(skillSnapshotHash(await store.files(companyId, id)), skillSnapshotHash(files)); + await fs.rm(skillPackagePaths(companyId, id).directory, { recursive: true }); + assert.equal(skillSnapshotHash(await store.files(companyId, id)), skillSnapshotHash(files)); + assert.equal(await store.verifyDurable(companyId, id), true); + console.log(JSON.stringify({ provider: "s3", files: files.length, archiveBytes: receipt.archiveBytes, + decodedBytes: receipt.totalBytes, multipartRoundTrip: true, coldRestore: true, verifiedDurable: true })); +} finally { + await Promise.allSettled(keys.map(key => storage.deleteObject(companyId, key))); + if (bucketCreated) await client.send(new DeleteBucketCommand({ Bucket: bucket })).catch(error => console.error("Disposable bucket cleanup failed", error)); + client.destroy(); + await db.$client.end({ timeout: 1 }); + await database.cleanup(); + await fs.rm(home, { recursive: true, force: true }); + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; +} diff --git a/server/src/__tests__/company-skills-service.test.ts b/server/src/__tests__/company-skills-service.test.ts index 08961fe7ec6..5d7f1675048 100644 --- a/server/src/__tests__/company-skills-service.test.ts +++ b/server/src/__tests__/company-skills-service.test.ts @@ -53,7 +53,9 @@ describeEmbeddedPostgres("companySkillService.list", () => { return skillDir; } + const previousPackageStorage = process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; beforeAll(async () => { + process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "true"; tempDb = await startEmbeddedPostgresTestDatabase("paperclip-company-skills-service-"); oldPaperclipHome = process.env.PAPERCLIP_HOME; oldPaperclipInstanceId = process.env.PAPERCLIP_INSTANCE_ID; @@ -80,6 +82,8 @@ describeEmbeddedPostgres("companySkillService.list", () => { }); afterAll(async () => { + if (previousPackageStorage === undefined) delete process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; + else process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = previousPackageStorage; if (oldPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = oldPaperclipHome; if (oldPaperclipInstanceId === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; @@ -233,7 +237,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { await db.insert(companies).values({ id: companyId, name: "Rollback retry", issuePrefix: `T${companyId.slice(0, 6)}` }); const input = { name: "Retryable", slug: "retryable", markdown: "# durable bytes\n" }; await expect(db.transaction(async (tx) => { - await companySkillService(tx as any).createLocalSkill(companyId, input); + await companySkillService(tx as any, { packageDb: db }).createLocalSkill(companyId, input); throw new Error("simulate outer rollback"); })).rejects.toThrow("simulate outer rollback"); @@ -383,7 +387,7 @@ describeEmbeddedPostgres("companySkillService.list", () => { await db.insert(companies).values({ id: companyId, name: "Rollback conflict", issuePrefix: `T${companyId.slice(0, 6)}` }); const original = { name: "Original", slug: "rollback-conflict", markdown: "# original bytes\n" }; await expect(db.transaction(async (tx) => { - await companySkillService(tx as any).createLocalSkill(companyId, original); + await companySkillService(tx as any, { packageDb: db }).createLocalSkill(companyId, original); throw new Error("simulate outer rollback"); })).rejects.toThrow("simulate outer rollback"); diff --git a/server/src/__tests__/skill-package-store.test.ts b/server/src/__tests__/skill-package-store.test.ts new file mode 100644 index 00000000000..088949a9122 --- /dev/null +++ b/server/src/__tests__/skill-package-store.test.ts @@ -0,0 +1,336 @@ +import { skillPackageWork } from "../services/skill-package-work.js"; +import * as recovery from "../services/skill-package-recovery.js"; +import type { GitHubRead } from "../services/github-skill-source.js"; +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { and, eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { companies, companySkills, companySkillVersions, companySkillRepositorySnapshots as snapshots, + skillPackageDeletions, createDb } from "@paperclipai/db"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import * as storageModule from "../storage/index.js"; +import { createStorageService } from "../storage/service.js"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { companySkillService } from "../services/company-skills.js"; +import { skillPackageStore, skillPackagePaths, withSkillPackageReadScope } from "../services/skill-package-store.js"; +import { skillPackageMaintenance } from "../services/skill-package-maintenance.js"; +import { skillPackageManifest, validateSkillPackageManifest, writeSkillPackageArchive, extractSkillPackageArchive, + verifySkillPackageDirectory } from "../services/skill-package-files.js"; +import { skillFileBytes, skillSnapshotHash } from "../services/skill-snapshot.js"; +import { hasRestorableSnapshot } from "../services/skill-version-cache.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("external skill packages", () => { + let database: Awaited>; + let db: ReturnType; + let home: string; + let storage: storageModule.StorageService; + const previousHome = process.env.PAPERCLIP_HOME; + const files: CompanySkillVersionFileInventoryEntry[] = [ + { path: "skills/one/SKILL.md", kind: "skill", content: "---\nname: one\ndescription: Test one\n---\nUse ../../shared/run.sh\n" }, + { path: "skills/two/SKILL.md", kind: "skill", content: "---\nname: two\ndescription: Test two\n---\nUse ../../shared/run.sh\n" }, + { path: "shared/run.sh", kind: "script", content: "#!/bin/sh\necho original\n", executable: true }, + { path: "assets/binary", kind: "reference", content: Buffer.from([0, 255, 7]).toString("base64"), encoding: "base64" }, + { path: "assets/base64-text", kind: "reference", content: Buffer.from("text encoded explicitly").toString("base64"), encoding: "base64" }, + ]; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("skill-packages-"); + db = createDb(database.connectionString, { maxConnections: 1 }); + home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-packages-")); + process.env.PAPERCLIP_HOME = home; + storage = createStorageService(createLocalDiskStorageProvider(path.join(home, "objects"))); + }); + afterEach(() => vi.restoreAllMocks()); + afterAll(async () => { + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + await db?.$client.end({ timeout: 1 }); + await database?.cleanup(); + if (home) await fs.rm(home, { recursive: true, force: true }); + }); + async function company() { + const id = randomUUID(); + await db.insert(companies).values({ id, name: "Package test", issuePrefix: `P${id.slice(0, 6)}` }); + return id; + } + async function fixture() { + const companyId = await company(); + vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage); + const store = skillPackageStore(db, storage); + const id = await store.prepare(companyId, files); + const skills = []; + for (const name of ["one", "two"]) { + const [skill] = await db.insert(companySkills).values({ companyId, key: `company/${companyId}/${name}`, slug: name, + name, markdown: files.find(file => file.path === `skills/${name}/SKILL.md`)!.content, sourceType: "github", + metadata: { skillSourceId: randomUUID(), snapshotHash: "saved" } }).returning(); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 1, + repositorySnapshotId: id, repositorySkillPath: `skills/${name}/SKILL.md`, packageSnapshotId: id, + packageFileMap: [{ path: "SKILL.md", kind: "skill", sourcePath: `skills/${name}/SKILL.md` }] }).returning(); + await db.update(companySkills).set({ currentVersionId: version!.id }).where(eq(companySkills.id, skill!.id)); + skills.push({ skill: skill!, version: version! }); + } + const service = companySkillService(db); + const resolve = () => service.listRuntimeSkillEntries(companyId, { onlyKeys: new Set(skills.map(({ skill }) => skill.key)), snapshotLatest: true }); + return { companyId, id, store, skills, service, resolve }; + } + + it("automatically converts legacy rows, clears verified payloads and preserves version identities", async () => { + const companyId = await company(); + const [snapshot] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const [skill] = await db.insert(companySkills).values({ companyId, key: "legacy", slug: "legacy", name: "Legacy", markdown: files[0]!.content }).returning(); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 7, + repositorySnapshotId: snapshot!.id, repositorySkillPath: files[0]!.path, + fileInventory: [{ ...files[0]!, path: "SKILL.md" }] }).returning(); + const work = skillPackageWork(db, storage); + await work.tick(new AbortController().signal); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.id, snapshot!.id)); + const [converted] = await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version!.id)); + expect(saved!.fileInventory).toEqual([]); + expect(saved!.migrationAt).toBeNull(); + expect(saved!.storage).not.toBeNull(); + expect(converted!.fileInventory).toEqual([]); + expect(converted!.migrationAt).toBeNull(); + expect(converted!.packageSnapshotId).toBe(snapshot!.id); + expect(converted!.revisionNumber).toBe(7); + expect(await skillPackageStore(db, storage).versionFiles(converted!)).toEqual([{ ...files[0]!, path: "SKILL.md" }]); + }); + + it("expires an unused Git package and reacquires its exact bytes without a migration command", async () => { + const f = await fixture(); + const recipe = { version: 1 as const, sourceId: randomUUID(), repositoryId: "123", repositoryUrl: "https://github.com/acme/skills", + commitSha: "a".repeat(40), skillPath: null, manifest: skillPackageManifest(files) }; + await f.store.prepare(f.companyId, files, true, undefined, recipe); + await f.store.files(f.companyId, f.id); + await db.update(snapshots).set({ createdAt: new Date(Date.now() - 40 * 86_400_000), maintenanceAt: new Date(0), migrationAt: null }).where(eq(snapshots.id, f.id)); + // Current versions stay pinned even after their cache age expires. + await skillPackageWork(db, storage).tick(new AbortController().signal); + expect((await f.store.descriptor(f.companyId, f.id)).storage).not.toBeNull(); + for (const { skill } of f.skills) await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.id, skill.id)); + await db.update(snapshots).set({ maintenanceAt: new Date(0) }).where(eq(snapshots.id, f.id)); + await skillPackageWork(db, storage).tick(new AbortController().signal); + expect((await f.store.descriptor(f.companyId, f.id)).storage).toBeNull(); + expect(await fs.stat(skillPackagePaths(f.companyId, f.id).directory).catch(() => null)).toBeNull(); + const entries = recipe.manifest.files.map((file, i) => ({ path: file.path, mode: file.executable ? "100755" : "100644", type: "blob", size: file.size, sha: String(i).repeat(40) })); + const read = Object.assign(vi.fn(async () => ({ id: 123 })), { openSnapshot: vi.fn(async () => ({ commitSha: recipe.commitSha, entries, + readBlob: async (entry: typeof entries[number]) => skillFileBytes(files[entries.indexOf(entry)]!), release: async () => {} })) }) as unknown as GitHubRead; + const authorization = vi.spyOn(recovery, "skillPackageRecoveryReader").mockResolvedValue(read); + expect(await f.store.files(f.companyId, f.id)).toEqual(files); + expect(read.openSnapshot).toHaveBeenCalledOnce(); + await f.store.files(f.companyId, f.id); + expect(authorization).toHaveBeenCalledOnce(); + await fs.rm(skillPackagePaths(f.companyId, f.id).directory, { recursive: true }); + authorization.mockRejectedValue(new Error("grant revoked")); + await expect(f.store.files(f.companyId, f.id)).rejects.toThrow("grant revoked"); + await expect(f.store.files(f.companyId, f.id)).rejects.toThrow("backoff"); + expect(authorization).toHaveBeenCalledTimes(2); + }); + + it("recovers missing archive and manifest objects while a published receipt still exists", async () => { + const companyId = await company(); + const store = skillPackageStore(db, storage); + const recipe = { version: 1 as const, sourceId: randomUUID(), repositoryId: "123", repositoryUrl: "https://github.com/acme/skills", + commitSha: "a".repeat(40), skillPath: null, manifest: skillPackageManifest(files) }; + const id = await store.prepare(companyId, files, true, undefined, recipe); + const receipt = (await store.descriptor(companyId, id)).storage!; + await storage.deleteObject(companyId, receipt.archiveKey); + await storage.deleteObject(companyId, receipt.manifestKey); + await expect(recovery.skillPackageRecoveryReader(db, companyId, recipe)).rejects.toThrow("Reconnect its original repository"); + const entries = recipe.manifest.files.map((file, i) => ({ path: file.path, mode: file.executable ? "100755" : "100644", type: "blob", size: file.size, sha: String(i).repeat(40) })); + const read = Object.assign(vi.fn(async () => ({ id: 123 })), { openSnapshot: vi.fn(async () => ({ commitSha: recipe.commitSha, entries, + readBlob: async (entry: typeof entries[number]) => skillFileBytes(files[entries.indexOf(entry)]!), release: async () => {} })) }) as unknown as GitHubRead; + vi.spyOn(recovery, "skillPackageRecoveryReader").mockResolvedValue(read); + // PostgreSQL JSONB has reordered recipe object keys; its canonical bytes must + // still agree with the original externally written manifest checksum. + expect(await store.files(companyId, id)).toEqual(files); + expect(read.openSnapshot).toHaveBeenCalledOnce(); + }); + + it("retains referenced historical bytes without an exact Git recipe", async () => { + const f = await fixture(); + for (const { skill } of f.skills) await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.id, skill.id)); + await db.update(snapshots).set({ createdAt: new Date(0), migrationAt: null, maintenanceAt: new Date(0) }).where(eq(snapshots.id, f.id)); + await skillPackageWork(db, storage).tick(new AbortController().signal); + expect((await f.store.descriptor(f.companyId, f.id)).storage).not.toBeNull(); + expect(await f.store.files(f.companyId, f.id)).toEqual(files); + }); + + it("restores one shared archive, then prepares a warm turn without object reads or SQL file bodies", async () => { + const f = await fixture(); + const read = vi.spyOn(storage, "getObject"); + const cold = await f.resolve(); + expect(cold.map(entry => entry.sourceStatus)).toEqual(["available", "available"]); + expect(read.mock.calls.filter(call => call[1].endsWith("files.tar.gz"))).toHaveLength(1); + expect(read.mock.calls.filter(call => call[1].endsWith("manifest.json"))).toHaveLength(1); + const [row] = await db.select().from(snapshots).where(eq(snapshots.id, f.id)); + expect(row!.fileInventory).toEqual([]); + const sql: string[] = []; + const oldDebug = db.$client.options.debug; + db.$client.options.debug = (_connection, query) => { sql.push(query); }; + read.mockClear(); + try { expect(await f.resolve()).toEqual(cold); } + finally { db.$client.options.debug = oldDebug; } + expect(read).not.toHaveBeenCalled(); + expect(sql.length).toBeLessThanOrEqual(6); + for (const query of sql.filter(query => /from "company_skill_(repository_snapshots|versions)"/.test(query))) { + // Bundled release validation returns only a DB-side digest. + if (!query.includes("encode(sha256(")) expect(query).not.toMatch(/"file_inventory"/); + } + expect((await f.service.getVersion(f.companyId, f.skills[0]!.skill.id, f.skills[0]!.version.id))!.fileInventory) + .toEqual([{ path: "SKILL.md", kind: "skill", content: files[0]!.content }]); + const restored = await f.store.files(f.companyId, f.id); + expect(restored.map(skillFileBytes)).toEqual(files.map(skillFileBytes)); + await expect(f.store.files(await company(), f.id)).rejects.toThrow("unavailable"); + }); + + it("repairs a same-size cached edit and restores old versions without upstream access", async () => { + const f = await fixture(); + const entries = await f.resolve(); + const target = path.join(entries[0]!.source, ".paperclip-repository/shared/run.sh"); + const stat = await fs.stat(target); + await fs.chmod(target, 0o755); + await fs.writeFile(target, "#!/bin/sh\necho modified\n"); + await fs.utimes(target, stat.atime, stat.mtime); + const read = vi.spyOn(storage, "getObject"); + await f.resolve(); + expect(await fs.readFile(target, "utf8")).toBe(files[2]!.content); + expect(read).not.toHaveBeenCalled(); + const upstream = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("offline")); + for (const entry of entries) await fs.rm(entry.source, { recursive: true }); + await fs.rm(skillPackagePaths(f.companyId, f.id).directory, { recursive: true }); + expect((await f.resolve()).every(entry => entry.sourceStatus === "available")).toBe(true); + expect(upstream).not.toHaveBeenCalled(); + }); + + it("refuses cache eviction when durable objects are missing, even after an earlier verification", async () => { + const f = await fixture(); + await f.resolve(); + expect(await hasRestorableSnapshot(db, f.companyId, f.skills[0]!.version.id)).toBe(true); + const receipt = (await f.store.descriptor(f.companyId, f.id)).storage!; + await storage.deleteObject(f.companyId, receipt.archiveKey); + await expect(hasRestorableSnapshot(db, f.companyId, f.skills[0]!.version.id)).rejects.toThrow(); + // Existing local verified files remain usable while storage is unavailable. + expect((await f.resolve()).every(entry => entry.sourceStatus === "available")).toBe(true); + }); + + it("retries expired upload claims with fresh keys and only deletes the abandoned attempt", async () => { + const companyId = await company(); + const store = skillPackageStore(db, storage); + const put = vi.spyOn(storage, "putFile").mockRejectedValueOnce(new Error("connection lost")); + await expect(store.prepare(companyId, files)).rejects.toThrow("connection lost"); + put.mockRestore(); + const [pending] = await db.select().from(snapshots).where(eq(snapshots.companyId, companyId)); + expect(pending!.storage).toBeNull(); + await expect(store.prepare(companyId, files)).rejects.toThrow("being saved"); + await db.update(snapshots).set({ upload: { ...pending!.upload!, expiresAt: new Date(0).toISOString() } }).where(eq(snapshots.id, pending!.id)); + const id = await store.prepare(companyId, files); + expect(id).toBe(pending!.id); + const receipt = (await store.descriptor(companyId, id)).storage!; + expect(receipt.archiveKey).not.toBe(pending!.upload!.archiveKey); + const [deletion] = await db.select().from(skillPackageDeletions).where(eq(skillPackageDeletions.companyId, companyId)); + expect(deletion!.archiveKey).toBe(pending!.upload!.archiveKey); + await db.update(skillPackageDeletions).set({ deleteAfter: new Date(0) }).where(eq(skillPackageDeletions.id, deletion!.id)); + expect((await skillPackageMaintenance(db, storage).deleteObjects({ companyId, apply: true })).removed).toBe(1); + expect(await store.verifyDurable(companyId, id)).toBe(true); + // Company cascades retain a receipt after removing the snapshot's DB row. + await db.delete(companies).where(eq(companies.id, companyId)); + expect(await db.select().from(skillPackageDeletions).where(eq(skillPackageDeletions.companyId, companyId))) + .toEqual([expect.objectContaining({ archiveKey: receipt.archiveKey })]); + }); + + it("cancels queued and active uploads without publishing or leaking I/O capacity", async () => { + const ids = await Promise.all([company(), company(), company()]); + const active = new AbortController(); + const queued = new AbortController(); + const put = vi.spyOn(storage, "putFile").mockImplementation(async input => { + expect(input.signal).toBeDefined(); + await new Promise((_resolve, reject) => { + input.signal!.addEventListener("abort", () => reject(input.signal!.reason), { once: true }); + }); + throw new Error("Unreachable cancelled upload"); + }); + const store = skillPackageStore(db, storage); + const running = ids.slice(0, 2).map(id => store.prepare(id!, files, false, active.signal).catch(error => error)); + try { + await vi.waitFor(() => expect(put).toHaveBeenCalledTimes(2)); + const waiting = store.prepare(ids[2]!, files, false, queued.signal).catch(error => error); + await vi.waitFor(async () => expect(await db.select({ id: snapshots.id }).from(snapshots) + .where(eq(snapshots.companyId, ids[2]!))).toHaveLength(1)); + queued.abort(new Error("queued cancellation")); + expect((await waiting).message).toBe("queued cancellation"); + expect(put).toHaveBeenCalledTimes(2); + } finally { + active.abort(new Error("active cancellation")); + await Promise.all(running); + put.mockRestore(); + } + for (const id of ids) { + const [row] = await db.select().from(snapshots).where(eq(snapshots.companyId, id!)); + expect(row!.storage).toBeNull(); + expect(row!.upload).not.toBeNull(); + } + expect(await store.prepare(await company(), files)).toEqual(expect.any(String)); + }); + + it("migrates both legacy payload locations in bounded resumable pages without changing version IDs", async () => { + const companyId = await company(); + const [snapshot] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const [skill] = await db.insert(companySkills).values({ companyId, key: "legacy", slug: "legacy", name: "Legacy", markdown: files[0]!.content }).returning(); + const original = { path: "SKILL.md", kind: "skill" as const, content: files[0]!.content }; + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 7, + repositorySnapshotId: snapshot!.id, repositorySkillPath: files[0]!.path, fileInventory: [original] }).returning(); + const maintenance = skillPackageMaintenance(db, storage); + expect((await maintenance.migrateSnapshots({ companyId })).migrated).toBe(0); + const fail = vi.spyOn(storage, "getObject").mockRejectedValueOnce(new Error("verification interrupted")); + await expect(maintenance.migrateSnapshots({ companyId, apply: true })).rejects.toThrow("verification interrupted"); + fail.mockRestore(); + const [pending] = await db.select().from(snapshots).where(eq(snapshots.id, snapshot!.id)); + expect(pending!.fileInventory).toEqual(files); + await db.update(snapshots).set({ upload: { ...pending!.upload!, expiresAt: new Date(0).toISOString() } }).where(eq(snapshots.id, snapshot!.id)); + expect((await maintenance.migrateSnapshots({ companyId, limit: 1, apply: true })).migrated).toBe(1); + const [soaked] = await db.select().from(snapshots).where(eq(snapshots.id, snapshot!.id)); + expect(soaked!.storage).not.toBeNull(); + expect(soaked!.fileInventory).toEqual(files); + expect((await maintenance.migrateSnapshots({ companyId, limit: 1, apply: true, clearLegacy: true })).migrated).toBe(1); + expect((await maintenance.migrateSnapshots({ companyId, apply: true })).inspected).toBe(0); + expect((await maintenance.migrateVersions({ companyId, apply: true, clearLegacy: true })).migrated).toBe(1); + const [saved] = await db.select().from(companySkillVersions).where(and(eq(companySkillVersions.id, version!.id), eq(companySkillVersions.companyId, companyId))); + expect(saved).toMatchObject({ id: version!.id, revisionNumber: 7, fileInventory: [], repositorySnapshotId: snapshot!.id, packageSnapshotId: snapshot!.id }); + expect(await skillPackageStore(db, storage).versionFiles(saved!)).toEqual([original]); + }); + + it.each([false, true])("restores across legacy payload clearing with a cached manifest: %s", async cachedManifest => { + const companyId = await company(); + const [snapshot] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const store = skillPackageStore(db, storage); + await withSkillPackageReadScope(db, async () => { + expect((await store.descriptor(companyId, snapshot!.id)).storage).toBeNull(); + if (cachedManifest) await store.manifest(companyId, snapshot!.id); + await store.prepare(companyId, files, true); + await db.update(snapshots).set({ fileInventory: [] }).where(eq(snapshots.id, snapshot!.id)); + expect(skillSnapshotHash(await store.files(companyId, snapshot!.id))).toBe(skillSnapshotHash(files)); + expect(await store.verifyDurable(companyId, snapshot!.id)).toBe(true); + }); + }); + + it("bounds and validates archives, binaries, paths, executable modes, and corruption", async () => { + const root = await fs.mkdtemp(path.join(home, "codec-")); + const archive = path.join(root, "files.tar.gz"); + const manifest = skillPackageManifest(files); + await writeSkillPackageArchive(archive, files); + const output = path.join(root, "out"); + await fs.mkdir(output); + await extractSkillPackageArchive(archive, output, manifest); + expect(await verifySkillPackageDirectory(output, manifest.files)).toBe(true); + expect((await fs.stat(path.join(output, "shared/run.sh"))).mode & 0o111).toBe(0o111); + expect(() => validateSkillPackageManifest({ ...manifest, files: [{ ...manifest.files[0], path: "../escape" }] })).toThrow(); + expect(() => validateSkillPackageManifest({ ...manifest, files: [manifest.files[0], manifest.files[0]] })).toThrow(); + expect(() => validateSkillPackageManifest({ ...manifest, totalBytes: 1 })).toThrow(); + const bad = path.join(root, "bad"); + await fs.mkdir(bad); + await fs.truncate(archive, 20); + await expect(extractSkillPackageArchive(archive, bad, manifest)).rejects.toThrow(); + }); +}); diff --git a/server/src/__tests__/skill-sources.test.ts b/server/src/__tests__/skill-sources.test.ts index 7a8ca0d6043..c612ba60985 100644 --- a/server/src/__tests__/skill-sources.test.ts +++ b/server/src/__tests__/skill-sources.test.ts @@ -31,7 +31,9 @@ describe.skipIf(!support.supported)('skill source persistence', () => { let files: Record = initial; let commit = sha; const context: SkillSourceContext = { actor: { type: 'user', userId: 'board' }, read: () => githubFixture(files, { 'deep/one/scripts/run.sh': '100755', 'shared/check.sh': '100755' }, commit), authorize: async () => {}, audit: async () => {} }; + const previousPackageStorage = process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; beforeAll(async () => { + process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "true"; if (externalUrl) await applyPendingMigrations(externalUrl); else testDb = await startEmbeddedPostgresTestDatabase('paperclip-skill-sources-'); db = createDb(externalUrl ?? testDb.connectionString); @@ -39,10 +41,24 @@ describe.skipIf(!support.supported)('skill source persistence', () => { await db.insert(companies).values({ id: companyId, name: 'Skills', issuePrefix: 'SKL' }); }, 30000); afterAll(async () => { + if (previousPackageStorage === undefined) delete process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED; + else process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = previousPackageStorage; if (originalHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = originalHome; await db?.$client.end({ timeout: 1 }); await testDb?.cleanup(); if (home) await fs.rm(home, { recursive: true, force: true }); }); + it('tracks repositories with no selected valid skills without saving their contents', async () => { + for (const repository of [{ 'README.md': 'No skills here' }, { 'one/SKILL.md': md('one'), 'shared.txt': 'Unused package content' }]) { + const id = randomUUID(); + await db.insert(companies).values({ id, name: 'Discovery only', issuePrefix: `E${id.slice(0, 6)}` }); + const result = await skillSourceService(db).create(id, { repositoryUrl: 'https://github.com/acme/skills', trackingRef: 'main', + packageMode: 'repository', commitSha: sha, selectedPaths: [] }, { ...context, read: () => githubFixture(repository) }); + expect(result.imported).toEqual([]); + expect(result.source.lastScanCommit).toBe(sha); + expect(await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, id))).toEqual([]); + } + }); + it('publishes two assigned skills while a concurrent version save holds the second skill', async () => { const companyId = randomUUID(); await db.insert(companies).values({ id: companyId, name: 'Concurrent skill publication', issuePrefix: 'CSP' }); @@ -566,7 +582,7 @@ describe.skipIf(!support.supported)('skill source persistence', () => { const v1 = (await skills.getVersion(companyId, main.id, main.currentVersionId!))!; const helper = installed.imported.find(skill => skill.name === 'helper')!; expect((await skills.getVersion(companyId, helper.id, helper.currentVersionId!))?.repositorySnapshotId).toBe(v1.repositorySnapshotId); - expect(await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, companyId))).toHaveLength(1); + expect(await db.select().from(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.id, v1.repositorySnapshotId!))).toEqual([expect.objectContaining({ fileInventory: [], storage: expect.objectContaining({ format: "tar-gzip-v1" }) })]); await db.update(companySkills).set({ trustLevel: 'markdown_only' }).where(eq(companySkills.id, main.id)); expect((await sources.refresh(companyId, installed.source.id, context)).unchanged).toBe(2); expect((await skills.getById(companyId, main.id))?.trustLevel).toBe('scripts_executables'); diff --git a/server/src/index.ts b/server/src/index.ts index c7f7406b7a1..1de3d49028a 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -21,6 +21,8 @@ import { registerEnvironmentCleanupWork } from "./services/environment-cleanup-w import { loginCleanupSchedule } from "./services/login-cleanup-state.js"; import { registerLoginCleanupWork } from "./services/login-cleanup-work.js"; import { timeStartupRecovery } from "./startup-recovery-timing.js"; +import { runtimeAssetRetention } from "./services/native-runtime/runtime-asset-retention.js"; +import { skillPackageWork } from "./services/skill-package-work.js"; import { skillSourceSyncService } from "./services/skill-source-sync.js"; import { SKILL_VERSION_RETENTION_INTERVAL_MS, @@ -1396,6 +1398,9 @@ async function startServerWithDatabaseTeardown( const accountingWorker = registerCostAccountingWork(db, deliveryWork, accountingHeartbeat.reconcileCostAccounting); server.on("close", accountingWorker.stop); skillSourceSync.start(deliveryWork); + skillPackageWork(db).start(deliveryWork); + const runtimeAssetCleanup = runtimeAssetRetention(db).start(deliveryWork); + server.on("close", () => { void runtimeAssetCleanup.stop(); }); const caseHandoffDeliveries = caseHandoffDeliveryService(db as any, { heartbeat: environmentLeaseCleanupHeartbeat }); deliveryWork.register(DELIVERY_QUEUES.caseHandoffs, { retryMs: 5_000, diff --git a/server/src/routes/company-skills.ts b/server/src/routes/company-skills.ts index 93a396fcb9c..3926793700a 100644 --- a/server/src/routes/company-skills.ts +++ b/server/src/routes/company-skills.ts @@ -1221,7 +1221,7 @@ export function companySkillRoutes(db: Db) { eq(activityLog.companyId, companyId), eq(activityLog.action, "company.skill_created"), sql`${activityLog.details}->>'idempotencyKey' = ${receiptKey}`, )); - const service = companySkillService(tx as unknown as Db); + const service = companySkillService(tx as unknown as Db, { packageDb: db }); if (prior) { if (prior.details?.fingerprint !== fingerprint) throw conflict("Skill idempotency key was used with different inputs"); const skill = await service.getById(companyId, prior.entityId); @@ -1298,7 +1298,7 @@ export function companySkillRoutes(db: Db) { } let publication: ActivityPublication | null = null; let receipt: { skillId: string; path: string; versionId: string | null; studioPath: string } | null = null; - await companySkillService(tx as unknown as Db).updateFile(companyId, skillId, input.path, input.content, skillActor(req), { + await companySkillService(tx as unknown as Db, { packageDb: db }).updateFile(companyId, skillId, input.path, input.content, skillActor(req), { encoding: input.encoding, executable: input.executable, expectedVersionId, onRollback: (restore, versionId) => { rollbackState.current = { restore, versionId }; }, afterUpdate: async (versionId) => { diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index 819f2745abb..b227e460ca9 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -68,6 +68,7 @@ export async function deleteCompany(db: Db, id: string) { await notifyChatCompanyConfiguration(tx, id); await chatCompanyDeletion.deleteCompanyData(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); await notifyDeliveryWork(tx, DELIVERY_QUEUES.secretProposalExpiry, id); await notifyToolReviewChange(tx, id); await agentObservabilityService(tx as unknown as Db).removeForCompanyDeletion(id); diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index a0926193ce4..f8f1f44ae90 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -1,3 +1,8 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; +import { verifyFileManifest } from "./file-manifest-cache.js"; +import { skillPackageStorageEnabled, skillPackageStore, skillVersionMetadataColumns, withSkillPackageReadScope, type StoredSkillVersion } from "./skill-package-store.js"; +import { skillPackageRuntime } from "./skill-package-runtime.js"; +import type { SkillPackageFileMapping } from "@paperclipai/db"; import { ensureSkillCacheDirectory, touchSkillVersionCache, withSkillVersionCacheLock } from "./skill-version-cache.js"; import { measureRunPhase } from "./execution-preparation-timing.js"; import { verifyExistingBundledSkillRelease } from "./bundled-skill-release-validation.js"; @@ -24,7 +29,7 @@ import { companySkillTestInputs, companySkillTestRunTemplates, companySkillTestRuns, - companySkillVersions, companySkillRepositorySnapshots, + companySkillVersions, withDedicatedDbConnection, companySkills, costEvents, documents, @@ -406,6 +411,8 @@ type BundledSkillReleaseManifestEntry = { type CreateVersionOptions = { repositorySnapshotId?: string; repositorySkillPath?: string; + packageSnapshotId?: string; + packageFileMap?: SkillPackageFileMapping[] | null; fileInventory?: CompanySkillVersionFileInventoryEntry[]; release?: { id: string; name: string; releasedAt: Date }; updateCurrentVersion?: boolean; @@ -1929,8 +1936,9 @@ function serializeVersionFileInventory( } function toCompanySkillVersion(row: CompanySkillVersionRow): CompanySkillVersion { + const { packageSnapshotId: _package, packageFileMap: _map, migrationAt: _migration, ...publicRow } = row; return { - ...row, + ...publicRow, label: row.label ?? null, releaseId: row.releaseId ?? null, releaseName: row.releaseName ?? null, @@ -2964,7 +2972,10 @@ async function listLastEditorsBySkillId( return editors; } -export function companySkillService(db: Db, options: { actor?: AuthorizationActor } = {}) { +export function companySkillService(db: Db, options: { actor?: AuthorizationActor; packageDb?: Db } = {}) { + const packageDb = options.packageDb ?? db; + const packages = skillPackageStore(packageDb); + const packageRuntime = skillPackageRuntime(packageDb); const visibility = (database: DbOrTransaction = db) => skillAccessService(database as Db, options.actor); const folderSvc = folderService(db); const agents = agentService(db); @@ -3228,7 +3239,10 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo await removeRuntimeSkillCache(resolveManagedSkillsRoot(companyId), skill.id, async () => { await fs.rm(resolveRuntimeSkillMaterializedPath(companyId, skill), { recursive: true, force: true }); - await db.delete(companySkills).where(eq(companySkills.id, skill.id)); + await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); + await tx.delete(companySkills).where(eq(companySkills.id, skill.id)); + }); }); } } @@ -3382,12 +3396,12 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo return readFullSkillRows(companyId); } - async function readFullSkillRows(companyId: string): Promise { - if (!visibility().canReadCompany(companyId)) return []; + async function readFullSkillRows(companyId: string, keys?: Set, unrestrictedOnly = false): Promise { + if (!visibility().canReadCompany(companyId) || keys?.size === 0) return []; const rows = await db .select(selectCompanySkillColumns()) .from(companySkills) - .where(and(eq(companySkills.companyId, companyId), await visibility().condition(companyId))) + .where(and(eq(companySkills.companyId, companyId), keys ? inArray(companySkills.key, [...keys]) : undefined, unrestrictedOnly ? isNull(companySkills.accessSourceId) : await visibility().condition(companyId))) .orderBy(asc(companySkills.name), asc(companySkills.key)); return rows.map((row) => toCompanySkill(row)); } @@ -3462,18 +3476,17 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ?? await getByKey(companyId, ref); } + async function readVersionMetadata(companyId: string, skillId: string, versionId: string): Promise { + if (db === packageDb) return packages.version(companyId, skillId, versionId); + const [row] = await db.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.companySkillId, skillId), eq(companySkillVersions.id, versionId))); + return row ?? null; + } + async function getVersion(companyId: string, skillId: string, versionId: string): Promise { if (!await getById(companyId, skillId)) return null; - const row = await db - .select() - .from(companySkillVersions) - .where(and( - eq(companySkillVersions.companyId, companyId), - eq(companySkillVersions.companySkillId, skillId), - eq(companySkillVersions.id, versionId), - )) - .then((rows) => rows[0] ?? null); - return row ? toCompanySkillVersion(row) : null; + const row = await readVersionMetadata(companyId, skillId, versionId); + return row ? toCompanySkillVersion({ ...row, fileInventory: await packages.versionFiles(row) }) : null; } async function getCurrentVersion(skill: CompanySkill): Promise { @@ -3543,14 +3556,15 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash && skill.currentVersionId) { const versionId = skill.currentVersionId; return withSkillVersionCacheLock(db, companyId, async () => { - const version = await getVersion(companyId, skill.id, versionId); + const version = await readVersionMetadata(companyId, skill.id, versionId); if (!version) throw unprocessable("Installed skill version is unavailable."); const sourceLocator = await materializeVersionSnapshot(companyId, skill, version); const audit = await auditInstalledSkillBytes({ ...skill, sourceType: "local_path", sourceLocator, - fileInventory: await runtimeVersionInventory(companyId, version), + fileInventory: version.repositorySnapshotId || version.packageSnapshotId + ? await packageRuntime.inventory(version) : await packages.versionFiles(version), }); await persistAuditMetadata(skill, audit); return audit; @@ -3619,7 +3633,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const existingForks = await existingForkSummaries(companyId, skill.id, actor); const currentVersion = await getCurrentVersion(skill); const repositoryInventory = currentVersion?.repositorySnapshotId - ? await runtimeVersionInventory(companyId, currentVersion) : null; + ? await packageRuntime.inventory((await readVersionMetadata(companyId, skill.id, currentVersion.id))!) : null; return enrichSkill( repositoryInventory ? { ...skill, fileInventory: repositoryInventory.map(({ path, kind }) => ({ path, kind })) } : skill, usedByAgents.length, @@ -3671,11 +3685,13 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const skill = await getById(companyId, skillId); if (!skill) throw notFound("Skill not found"); const rows = await db - .select() + .select(skillVersionMetadataColumns) .from(companySkillVersions) .where(and(eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.companySkillId, skillId))) .orderBy(desc(companySkillVersions.revisionNumber)); - return rows.map((row) => toCompanySkillVersion(row)); + const result: CompanySkillVersion[] = []; + for (const row of rows) result.push(toCompanySkillVersion({ ...row, fileInventory: await packages.versionFiles(row) })); + return result; } async function createVersion( @@ -3692,7 +3708,13 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo options.fileInventory ?? await collectVersionFileInventory(companyId, skill), ); const previousVersion = options.fileInventory === undefined ? await getCurrentVersion(skill) : null; + // Shipped release rows retain their existing validation contract. All user + // snapshots use the package store; source imports prepare before their transaction. + const packageSnapshotId = options.packageSnapshotId ?? (options.release || !skillPackageStorageEnabled() ? null : options.database + ? await withDedicatedDbConnection(packageDb, connection => skillPackageStore(connection).prepare(companyId, fileInventory)) + : await packages.prepare(companyId, fileInventory)); const persist = async (tx: DbOrTransaction) => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); await tx.execute(sql` select ${companySkills.id} from ${companySkills} @@ -3713,12 +3735,15 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo companySkillId: skillId, repositorySnapshotId: options.repositorySnapshotId ?? previousVersion?.repositorySnapshotId ?? null, repositorySkillPath: options.repositorySkillPath ?? previousVersion?.repositorySkillPath ?? null, + packageSnapshotId, + migrationAt: packageSnapshotId || options.release ? null : new Date(), + packageFileMap: options.packageFileMap ?? null, revisionNumber: Number(nextRevision ?? 1), label: input.label?.trim() || null, releaseId: options.release?.id ?? null, releaseName: options.release?.name ?? null, releasedAt: options.release?.releasedAt ?? null, - fileInventory, + fileInventory: packageSnapshotId ? [] : fileInventory, authorAgentId: actor?.type === "agent" ? actor.agentId ?? null : null, authorUserId: actor?.type === "user" ? actor.userId ?? null : null, }) @@ -3735,7 +3760,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo }; const versionRow = options.database ? await persist(options.database) : await db.transaction(persist); if (!versionRow) throw notFound("Failed to persist skill version"); - return toCompanySkillVersion(versionRow); + return toCompanySkillVersion({ ...versionRow, fileInventory }); } async function refreshStarCount(companyId: string, skillId: string) { @@ -4002,6 +4027,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo async function cleanupFailedFork(companyId: string, sourceSkillId: string, forkSkillId: string, forkDir: string) { await db.transaction(async (tx) => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); await tx .update(companySkills) .set({ currentVersionId: null, updatedAt: new Date() }) @@ -4407,11 +4433,19 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo async function readLoadedSkillFile(skill: CompanySkill, relativePath: string): Promise { const normalizedPath = normalizePortablePath(relativePath || "SKILL.md"); if (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash && skill.currentVersionId) { - const version = await getCurrentVersion(skill); - // Keep original package-relative reads working, and expose the repository - // files from the same saved snapshot used by runtime delivery. - const file = version?.fileInventory.find(entry => entry.path === normalizedPath) - ?? (version?.repositorySnapshotId ? (await runtimeVersionInventory(skill.companyId, version)).find(entry => entry.path === normalizedPath) : undefined); + const version = await readVersionMetadata(skill.companyId, skill.id, skill.currentVersionId); + let file: CompanySkillVersionFileInventoryEntry | null | undefined; + if (version?.packageSnapshotId) { + const mapping = version.packageFileMap?.find(file => file.path === normalizedPath); + if (mapping) file = await packages.withPackage(skill.companyId, version.packageSnapshotId, async (root, manifest) => { + const entry = manifest.files.find(file => file.path === mapping.sourcePath); + if (!entry) return null; + return { path: mapping.path, kind: mapping.kind, content: (await fs.readFile(path.join(root, entry.path))).toString(entry.encoding), + encoding: entry.encoding, executable: entry.executable }; + }); + else if (!version.repositorySnapshotId) file = await packageRuntime.readFile(version, normalizedPath); + } else if (version) file = (await packages.versionFiles(version)).find(file => file.path === normalizedPath); + if (!file && version?.repositorySnapshotId) file = await packageRuntime.readFile(version, normalizedPath); if (!file) throw notFound("Installed skill snapshot is missing a file."); return { ...file, skillId: skill.id, language: file.encoding === "base64" ? null : inferLanguageFromPath(normalizedPath), markdown: file.encoding !== "base64" && isMarkdownPath(normalizedPath), editable: false }; @@ -6018,15 +6052,13 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } async function runtimeVersionInventory(companyId: string, version: CompanySkillVersion) { - if (!version.repositorySnapshotId) return version.fileInventory; - const snapshotId = version.repositorySnapshotId; - const [snapshot] = await measureRunPhase("skills.repository.read", () => db.select().from(companySkillRepositorySnapshots).where(and( - eq(companySkillRepositorySnapshots.companyId, companyId), eq(companySkillRepositorySnapshots.id, snapshotId)))); - if (!snapshot || !version.repositorySkillPath) throw unprocessable("Installed repository snapshot is unavailable."); - return repositorySkillProjection(snapshot.fileInventory, version.repositorySkillPath); + if (!version.repositorySnapshotId) return version.fileInventory.length ? version.fileInventory + : packages.versionFiles((await readVersionMetadata(companyId, version.companySkillId, version.id))!); + if (!version.repositorySkillPath) throw unprocessable("Installed repository snapshot is unavailable."); + return repositorySkillProjection(await packages.files(companyId, version.repositorySnapshotId), version.repositorySkillPath); } - async function materializeVersionSnapshot(companyId: string, skill: CompanySkill, version: CompanySkillVersion): Promise { + async function materializeVersionSnapshot(companyId: string, skill: CompanySkill, version: StoredSkillVersion): Promise { return withSkillVersionCacheLock(db, companyId, async () => { const directory = await materializeVersionSnapshotLocked(companyId, skill, version); await touchSkillVersionCache(companyId, skill.id, version.id); @@ -6034,11 +6066,12 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo }); } - async function materializeVersionSnapshotLocked(companyId: string, skill: CompanySkill, version: CompanySkillVersion): Promise { - version = { ...version, fileInventory: await runtimeVersionInventory(companyId, version) }; + async function materializeVersionSnapshotLocked(companyId: string, skill: CompanySkill, version: StoredSkillVersion): Promise { const runtimeRoot = path.resolve(resolveManagedSkillsRoot(companyId), "__versions__", skill.id); const skillDir = path.join(runtimeRoot, version.id); return runVersionSnapshotFlight(skillDir, async () => { + if (version.repositorySnapshotId || version.packageSnapshotId) return packageRuntime.materialize(version, skillDir); + version = { ...version, fileInventory: await packages.versionFiles(version) }; await ensureSkillCacheDirectory(runtimeRoot, true); if (await measureRunPhase("skills.version.verify", () => materializedVersionSnapshotMatches(skillDir, version), { fileCount: version.fileInventory.length })) return skillDir; const staging = await fs.mkdtemp(path.join(runtimeRoot, `.${version.id}-`)); @@ -6081,7 +6114,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ?? (options.snapshotLatest || (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash) || skill.accessSourceId ? skill.currentVersionId : null); if (selectedVersionId) { const versionPath = path.resolve(resolveManagedSkillsRoot(companyId), "__versions__", skill.id, selectedVersionId); - const version = await measureRunPhase("skills.version.read", () => getVersion(companyId, skill.id, selectedVersionId)); + const version = await measureRunPhase("skills.version.read", () => readVersionMetadata(companyId, skill.id, selectedVersionId)); if (!version) { return { status: "missing", @@ -6185,22 +6218,43 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ): Promise { if (options.onlyKeys?.size === 0 || (!options.onlyKeys && options.selectionConfig && !options.includeLegacy && readPaperclipSkillSyncPreference(options.selectionConfig).desiredSkills.length === 0)) return consume([]); - await measureRunPhase("skills.inventory", () => ensureSkillInventoryCurrent(companyId)); - return withSkillVersionCacheLock(db, companyId, async () => { - const entries = await listRuntimeSkillEntriesLocked(companyId, options); + // Installed repository versions are immutable and already discovered. Do + // not reconcile the entire library (bundled releases/folders/local paths) + // before every turn. This probe caches no authority or mutable selection. + const requested = options.onlyKeys ?? (options.selectionConfig && !options.includeLegacy + ? new Set(readPaperclipSkillSyncPreference(options.selectionConfig).desiredSkills) : null); + const installed = requested?.size && visibility().canReadCompany(companyId) ? await db.select({ + key: companySkills.key, sourceType: companySkills.sourceType, versionId: companySkills.currentVersionId, accessSourceId: companySkills.accessSourceId, + }).from(companySkills).where(and(eq(companySkills.companyId, companyId), inArray(companySkills.key, [...requested]))) : []; + const preparedRepositorySelection = requested && installed.length === requested.size + && installed.every(skill => skill.sourceType === "github" && skill.versionId); + if (!preparedRepositorySelection) await measureRunPhase("skills.inventory", () => ensureSkillInventoryCurrent(companyId)); + return withSkillPackageReadScope(db, () => withSkillVersionCacheLock(db, companyId, async () => { + // Canonical keys already resolved by the fresh probe avoid listing all + // unrelated library references. Scope is still enforced by the row query; + // a public-to-project change between the two reads fails closed. + const runtimeOptions = preparedRepositorySelection ? { ...options, onlyKeys: requested } : options; + const entries = await listRuntimeSkillEntriesLocked(companyId, runtimeOptions, + Boolean(preparedRepositorySelection && installed.every(skill => skill.accessSourceId === null))); return measureRunPhase("skills.consume", () => consume(entries)); - }); + })); } async function listRuntimeSkillEntriesLocked( companyId: string, options: RuntimeSkillEntryOptions, + unrestrictedOnly = false, ): Promise { - const skills = await measureRunPhase("skills.rows", () => readFullSkillRows(companyId)); + const references = !options.onlyKeys && options.selectionConfig ? await listReferenceTargets(companyId) : []; const selectedKeys = options.onlyKeys ?? (options.selectionConfig ? new Set( (options.includeLegacy ? resolveLegacyPaperclipDesiredSkillNames : resolvePaperclipDesiredSkillNames)( - options.selectionConfig, skills.map(skill => ({ key: skill.key, runtimeName: buildSkillRuntimeName(skill.key, skill.slug) }))), - ) : null); + options.selectionConfig, references.map(skill => ({ key: skill.key, runtimeName: buildSkillRuntimeName(skill.key, skill.slug) }))), + ) : undefined); + const skills = await measureRunPhase("skills.rows", () => readFullSkillRows(companyId, selectedKeys, unrestrictedOnly)); + await packages.prefetch(companyId, skills.flatMap(skill => { + const id = options.versionSelections?.get(skill.key) ?? skill.currentVersionId; + return id ? [id] : []; + })); const out: PaperclipSkillEntry[] = []; for (let skill of skills) { @@ -6209,7 +6263,12 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const unpinned = !options.versionSelections?.get(skill.key); const installedDirectory = options.snapshotLatest && unpinned && (skill.sourceType === "local_path" || skill.sourceType === "catalog") ? await resolveExistingSkillDirectory(normalizeSkillDirectory(skill)) : null; - if (options.snapshotLatest && unpinned && (!skill.currentVersionId || installedDirectory)) { + const installedVersion = installedDirectory && skill.currentVersionId + ? await readVersionMetadata(companyId, skill.id, skill.currentVersionId) : null; + const installedUnchanged = installedDirectory && installedVersion?.packageSnapshotId && !installedVersion.packageFileMap + ? await verifyFileManifest(installedDirectory, (await packages.manifest(companyId, installedVersion.packageSnapshotId)).files) + : false; + if (!installedUnchanged && options.snapshotLatest && unpinned && (!skill.currentVersionId || installedDirectory)) { // Capture installed bytes only. Older remote imports can have no // version yet; execution must not fetch their missing files upstream. const installed: RuntimeSkillSourceResolution | null = installedDirectory ? { status: "available", source: installedDirectory } @@ -6219,8 +6278,11 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo : skill.fileInventory.length === 1 && skill.fileInventory[0]?.path === "SKILL.md" && skill.markdown ? [{ path: "SKILL.md", kind: "skill" as const, content: skill.markdown }] : null; if (inventory) { - const previous = skill.currentVersionId ? await getVersion(companyId, skill.id, skill.currentVersionId) : null; - if (!previous || skillSnapshotHash(previous.fileInventory) !== skillSnapshotHash(inventory)) { + const previous = installedVersion ?? (skill.currentVersionId ? await readVersionMetadata(companyId, skill.id, skill.currentVersionId) : null); + const previousHash = previous?.packageSnapshotId && !previous.packageFileMap + ? (await packages.descriptor(companyId, previous.packageSnapshotId)).contentHash + : previous ? skillSnapshotHash(await packages.versionFiles(previous)) : null; + if (previousHash !== skillSnapshotHash(inventory)) { const version = await createVersion(companyId, skill.id, { label: "Execution snapshot" }, { type: "system" }, { skipInventoryRefresh: true, skill, fileInventory: inventory }); skill = { ...skill, currentVersionId: version.id }; @@ -7366,6 +7428,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo } try { return await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); await beforeDelete?.(tx); const deleted: CompanySkill[] = []; for (const before of initial) { diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index afe718a27ec..ded9ca85cbb 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -74,6 +74,8 @@ export const DELIVERY_QUEUES = { dotEvents: "dot-events", grokDispatch: "grok-dispatch", skillSourceSync: "skill-source-sync", + skillPackages: "skill-packages", + runtimeAssetRetention: "runtime-asset-retention", fastResponse: "fast-responses", feedback: "feedback-exports", chatCompletion: "chat-completions", diff --git a/server/src/services/file-manifest-cache.ts b/server/src/services/file-manifest-cache.ts new file mode 100644 index 00000000000..97c360701a3 --- /dev/null +++ b/server/src/services/file-manifest-cache.ts @@ -0,0 +1,74 @@ +import { createHash } from "node:crypto"; +import { constants, promises as fs } from "node:fs"; +import path from "node:path"; + +export type FileManifestEntry = { path: string; size: number; sha256: string; executable: boolean; mode?: number }; + +// Cache only identities/digests, not file contents. ctime prevents an ordinary +// writer from hiding a modification by restoring mtime. Every use still checks +// paths, file types, count and mode; changed files are hashed again. +const verifiedFiles = new Map(); +const MAX_VERIFIED_FILES = 128_000; +let activeFileIo = 0; +const waitingFileIo: Array<() => void> = []; +async function withFileIo(action: () => Promise): Promise { + if (activeFileIo >= 24) await new Promise(resolve => waitingFileIo.push(resolve)); + else activeFileIo++; + try { return await action(); } + finally { const next = waitingFileIo.shift(); if (next) next(); else activeFileIo--; } +} +/** Rename preserves file identities; carry verification forward to avoid + * re-reading every byte on the first warm turn after a cold materialization. */ +export function relocateFileManifestVerification(from: string, to: string, files: readonly FileManifestEntry[]) { + for (const file of files) { + const source = path.join(from, file.path); + const verified = verifiedFiles.get(source); + verifiedFiles.delete(source); + if (verified) verifiedFiles.set(path.join(to, file.path), verified); + } +} +export async function verifyFileManifest(directory: string, files: readonly FileManifestEntry[]): Promise { + const expected = new Map(files.map(file => [file.path, file])); + const seen = new Set(); + async function walk(relative: string): Promise { + const root = path.join(directory, relative); + const rootStat = await fs.lstat(root); + if (!rootStat.isDirectory() || rootStat.isSymbolicLink()) return false; + const entries = await fs.readdir(root, { withFileTypes: true }); + for (let offset = 0; offset < entries.length; offset += 24) { + const outcomes = await Promise.all(entries.slice(offset, offset + 24).map(async entry => { + const name = relative ? `${relative}/${entry.name}` : entry.name; + if (entry.isDirectory()) return walk(name); + const file = expected.get(name); + if (!file || !entry.isFile()) return false; + seen.add(name); + return withFileIo(async () => { + const target = path.join(directory, name); + const stat = await fs.lstat(target, { bigint: true }); + if (!stat.isFile() || stat.size !== BigInt(file.size) || Boolean(stat.mode & 0o111n) !== file.executable + || (file.mode !== undefined && Number(stat.mode & 0o555n) !== file.mode)) return false; + const signature = `${stat.dev}:${stat.ino}:${stat.size}:${stat.mtimeNs}:${stat.ctimeNs}:${stat.mode}`; + const previous = verifiedFiles.get(target); + if (previous?.signature === signature && previous.digest === file.sha256) return true; + const handle = await fs.open(target, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const before = await handle.stat({ bigint: true }); + if (before.ino !== stat.ino || before.ctimeNs !== stat.ctimeNs) return false; + const hash = createHash("sha256"); + for await (const chunk of handle.createReadStream({ autoClose: false })) hash.update(chunk); + if (hash.digest("hex") !== file.sha256) return false; + const after = await handle.stat({ bigint: true }); + if (after.ctimeNs !== before.ctimeNs || after.mtimeNs !== before.mtimeNs) return false; + if (verifiedFiles.size >= MAX_VERIFIED_FILES) verifiedFiles.delete(verifiedFiles.keys().next().value!); + verifiedFiles.set(target, { signature, digest: file.sha256 }); + return true; + } finally { await handle.close(); } + }); + })); + if (outcomes.some(result => !result)) return false; + } + return true; + } + try { return await walk("") && seen.size === expected.size; } + catch (error) { if (["ENOENT", "ENOTDIR", "ELOOP"].includes((error as NodeJS.ErrnoException).code ?? "")) return false; throw error; } +} diff --git a/server/src/services/github-skill-source.ts b/server/src/services/github-skill-source.ts index f348affd27e..fc2dde27627 100644 --- a/server/src/services/github-skill-source.ts +++ b/server/src/services/github-skill-source.ts @@ -105,6 +105,12 @@ async function scanRepository(input: SkillScanInput, providerRead: GitHubRead, o } const repositoryPaths = indexSkillPackagePaths(entries.map(entry => entry.path)); const roots = entries.filter(e => e.type === 'blob' && /(^|\/)skill\.md$/i.test(e.path) && ['100644', '100755'].includes(e.mode)); + if (!roots.length) { + return { treeSha: snapshot.treeSha, downloadBytes: snapshot.downloadBytes, scanBytes: 0, + packageMode: input.packageMode, connectionId: providerRead.connectionId ?? null, + repositoryId: String(repo.id), repositoryUrl: canonical.repositoryUrl, fullName: repo.full_name, + trackingRef, commitSha: commit.sha, defaultBranch: repo.default_branch, candidates: [], warnings: [], skills: [] }; + } const rootsByDirectory = new Map(); for (const root of roots) { const dir = path.posix.dirname(root.path); diff --git a/server/src/services/heartbeat/run-preparation.ts b/server/src/services/heartbeat/run-preparation.ts index c55c81fbddb..9223444447c 100644 --- a/server/src/services/heartbeat/run-preparation.ts +++ b/server/src/services/heartbeat/run-preparation.ts @@ -1,3 +1,4 @@ +import { skillPackageStore, skillVersionMetadataColumns } from "../skill-package-store.js"; import { skillAccessService } from "../skill-access.js"; import { buildReviewPathRecoveryDisposition } from "../recovery/review-path-recovery.js"; import type { AuthorizationActor } from "../authorization.js"; @@ -2959,7 +2960,7 @@ export function createHeartbeatRunPreparation(db: Db) { inputId: companySkillTestRuns.inputId, skillVersionId: companySkillTestRuns.skillVersionId, outputDocumentKey: companySkillTestRuns.outputDocumentKey, - fileInventory: companySkillVersions.fileInventory, + version: skillVersionMetadataColumns, revisionNumber: companySkillVersions.revisionNumber, label: companySkillVersions.label, }) @@ -2979,8 +2980,9 @@ export function createHeartbeatRunPreparation(db: Db) { ) .then((rows) => rows[0] ?? null); if (!row) return null; - const fileInventory = Array.isArray(row.fileInventory) - ? row.fileInventory.flatMap((entry) => { + const savedFiles = await skillPackageStore(db).versionFiles(row.version); + const fileInventory = Array.isArray(savedFiles) + ? savedFiles.flatMap((entry) => { if (!entry || typeof entry !== "object" || Array.isArray(entry)) return []; const record = entry as unknown as Record; diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index 0e7463b2191..daa093682d8 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -213,6 +213,11 @@ const WORK_CHECKS = [ // label requires its completion stamp and a cleared failure reason. `SELECT 1 FROM feedback_exports WHERE (status = 'local_only' OR (status = 'sent' AND exported_at IS NOT NULL AND failure_reason IS NULL)) IS NOT TRUE`, + // Conversion owes a verified durable copy before dropping legacy payloads. + // Cache/deletion retention times are earliest eligibility, allowed to wait for + // the next ingress wake; they are not a promised external alarm deadline. + `SELECT 1 FROM company_skill_repository_snapshots WHERE migration_at IS NOT NULL`, + `SELECT 1 FROM company_skill_versions WHERE migration_at IS NOT NULL`, `SELECT 1 FROM project_skill_sources b JOIN companies c ON c.id = b.company_id WHERE c.status = 'active' AND b.active AND b.source_id IS NULL`, `SELECT 1 FROM company_skill_sources s JOIN companies c ON c.id = s.company_id diff --git a/server/src/services/native-runtime/runtime-asset-retention.test.ts b/server/src/services/native-runtime/runtime-asset-retention.test.ts new file mode 100644 index 00000000000..2d8444c0184 --- /dev/null +++ b/server/src/services/native-runtime/runtime-asset-retention.test.ts @@ -0,0 +1,57 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; +import { materializeAsset } from "./runtime-context.js"; +import { removeRuntimeAssetTree, runtimeAssetRetention, RUNTIME_ASSET_RETENTION_MS } from "./runtime-asset-retention.js"; + +let home: string; +const originalHome = process.env.PAPERCLIP_HOME; +beforeEach(async () => { home = await fs.mkdtemp(path.join(os.tmpdir(), "runtime-asset-gc-")); process.env.PAPERCLIP_HOME = home; }); +afterEach(async () => { + await removeRuntimeAssetTree(home); + if (originalHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = originalHome; + vi.restoreAllMocks(); +}); +function database(rows: unknown[] = []) { + const select = vi.fn(() => ({ from: () => ({ where: () => ({ limit: async () => rows }) }) })); + return { db: { select } as unknown as Db, select }; +} +async function age(digest: string) { + const manifest = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets", "manifests", `${digest}.json`); + const old = new Date(Date.now() - RUNTIME_ASSET_RETENTION_MS - 1_000); + await fs.utimes(manifest, old, old); +} +describe("runtime asset retention", () => { + it("has no database work when empty/fresh, removes expired copies and rebuilds the same digest", async () => { + const { db, select } = database(); + const worker = runtimeAssetRetention(db); + await worker.tick(new AbortController().signal); + expect(worker.nextRunAt()).toBeNull(); + expect(select).not.toHaveBeenCalled(); + const files = [{ path: "SKILL.md", content: Buffer.from("immutable skill"), mode: 0o444 }]; + const asset = await materializeAsset(files); + await worker.tick(new AbortController().signal); + expect(worker.nextRunAt()).toBeGreaterThan(Date.now()); + expect(select).not.toHaveBeenCalled(); + await age(asset.digest); + await worker.tick(new AbortController().signal); + expect(select).toHaveBeenCalledOnce(); + expect(await fs.stat(asset.rootPath).catch(() => null)).toBeNull(); + expect(worker.nextRunAt()).toBeNull(); + await worker.tick(new AbortController().signal); + expect(select).toHaveBeenCalledOnce(); + expect(await materializeAsset(files)).toEqual(asset); + }); + it("keeps files when an unfinished run has no trustworthy input", async () => { + const asset = await materializeAsset([{ path: "SKILL.md", content: Buffer.from("pinned skill"), mode: 0o444 }]); + await age(asset.digest); + const { db } = database([{ id: "run", companyId: "company", agentId: "agent", input: null }]); + const worker = runtimeAssetRetention(db); + await worker.tick(new AbortController().signal); + expect((await fs.stat(asset.rootPath)).isDirectory()).toBe(true); + expect(worker.nextRunAt()).toBeNull(); + }); +}); diff --git a/server/src/services/native-runtime/runtime-asset-retention.ts b/server/src/services/native-runtime/runtime-asset-retention.ts new file mode 100644 index 00000000000..2e74881888b --- /dev/null +++ b/server/src/services/native-runtime/runtime-asset-retention.ts @@ -0,0 +1,145 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { and, eq, sql } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import { withDirectoryPublicationLock } from "@paperclipai/adapter-utils/workspace-restore-merge"; +import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; +import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; +import { ensureSkillCacheDirectory, skillRunNeedsFilesSql } from "../skill-version-cache.js"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; + +export const RUNTIME_ASSET_RETENTION_MS = 30 * 24 * 60 * 60_000; +const TOUCH_MS = 60 * 60_000; +const touched = new Map(); +const observers = new Set<() => void>(); +const root = () => path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets"); +function paths(digest: string) { + if (!/^[a-f0-9]{64}$/.test(digest)) throw new Error("Invalid runtime asset digest"); + return { bundle: path.join(root(), "bundles", digest), manifest: path.join(root(), "manifests", `${digest}.json`) }; +} + +/** Persist an hourly access lease, under the same filesystem lock as deletion. + * Warm turns inside that lease add no I/O or database calls. The long grace also + * protects the short interval between preparation and durable run pinning. */ +export async function withRuntimeAssetUse(digest: string, consume: (locked: boolean) => Promise, mutation = false): Promise { + const target = paths(digest); + const last = touched.get(target.manifest); + const fresh = last !== undefined && Date.now() - last < TOUCH_MS; + if (fresh && !mutation) return consume(false); + await ensureSkillCacheDirectory(path.dirname(target.bundle), true); + return withDirectoryPublicationLock(target.bundle, async () => { + const result = await consume(true); + if (!fresh && (await fs.lstat(target.manifest).catch(() => null))?.isFile()) { + const now = Date.now(); + await fs.utimes(target.manifest, new Date(now), new Date(now)); + if (touched.size >= 4096) touched.delete(touched.keys().next().value!); + touched.set(target.manifest, now); + for (const wake of observers) wake(); + } + return result; + }); +} + +async function protectedAssets(db: Db): Promise | null> { + const rows = await db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + input: sql`${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput'` }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.runtimeMode, "native"), skillRunNeedsFilesSql())).limit(1001); + if (rows.length > 1000) return null; + const pins = new Set(); + for (const row of rows) { + try { + const input = parseNativeExecutionInput(row.input); + if (input.binding.runId !== row.id || input.binding.companyId !== row.companyId || input.binding.agentId !== row.agentId) return null; + if (!("runtimeContext" in input)) return null; + for (const asset of [input.runtimeContext.instructions.bundle, ...input.runtimeContext.skills.map(skill => skill.bundle)]) pins.add(asset.digest); + } catch { return null; } // An unfinished/legacy input cannot prove absence. + } + return pins; +} +export async function removeRuntimeAssetTree(directory: string) { + const stat = await fs.lstat(directory).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (!stat) return; + if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Unsafe runtime asset directory"); + await fs.chmod(directory, 0o700); + for (const entry of await fs.readdir(directory, { withFileTypes: true })) { + if (entry.isDirectory()) await removeRuntimeAssetTree(path.join(directory, entry.name)); + else if (!entry.isFile()) throw new Error("Unsafe runtime asset entry"); + } + await fs.rm(directory, { recursive: true }); +} + +export function runtimeAssetRetention(db: Db) { + let next: number | null = null; + async function tick(signal: AbortSignal) { + next = null; + const directory = path.join(root(), "manifests"); + try { await ensureSkillCacheDirectory(directory); } + catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } + const candidates: Array<{ digest: string; at: number }> = []; + const seen = new Set(); + const manifests = await fs.readdir(directory, { withFileTypes: true }).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return []; throw error; + }); + for (const entry of manifests) { + signal.throwIfAborted(); + if (!entry.isFile() || !/^[a-f0-9]{64}\.json$/.test(entry.name)) continue; + const stat = await fs.lstat(path.join(directory, entry.name)).catch(() => null); + if (!stat?.isFile()) continue; + seen.add(entry.name.slice(0, -5)); + const at = stat.mtimeMs + RUNTIME_ASSET_RETENTION_MS; + if (at <= Date.now()) candidates.push({ digest: entry.name.slice(0, -5), at }); + else next = Math.min(next ?? Infinity, at); + } + // A crash between bundle rename and manifest publication can leave an + // unreferenced bundle. The same age, lock and run-pin checks apply. + const bundles = path.join(root(), "bundles"); + try { + await ensureSkillCacheDirectory(bundles); + for (const entry of await fs.readdir(bundles, { withFileTypes: true })) { + signal.throwIfAborted(); + if (!entry.isDirectory() || !/^[a-f0-9]{64}$/.test(entry.name) || seen.has(entry.name)) continue; + const stat = await fs.lstat(path.join(bundles, entry.name)).catch(() => null); + if (!stat?.isDirectory()) continue; + const at = stat.mtimeMs + RUNTIME_ASSET_RETENTION_MS; + if (at <= Date.now()) candidates.push({ digest: entry.name, at }); + else next = Math.min(next ?? Infinity, at); + } + } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } + if (!candidates.length) return; // No SQL merely to discover idle history. + const pins = await protectedAssets(db); + if (!pins) return; // Reconsider on the next asset use or startup, without polling. + const eligible = candidates.filter(entry => !pins.has(entry.digest)); + for (const { digest } of eligible.slice(0, 32)) { + signal.throwIfAborted(); + const target = paths(digest); + await ensureSkillCacheDirectory(path.dirname(target.bundle), true); + await withDirectoryPublicationLock(target.bundle, async () => { + const stat = await fs.lstat(target.manifest).catch(() => null) ?? await fs.lstat(target.bundle).catch(() => null); + if (!stat || (!stat.isFile() && !stat.isDirectory()) || stat.isSymbolicLink()) return; + const at = stat.mtimeMs + RUNTIME_ASSET_RETENTION_MS; + if (at > Date.now()) { next = Math.min(next ?? Infinity, at); return; } + await removeRuntimeAssetTree(target.bundle); + await fs.rm(target.manifest, { force: true }); + touched.delete(target.manifest); + }); + } + if (eligible.length > 32) next = Math.min(next ?? Infinity, Date.now() + 1_000); + } + function start(coordinator: Pick, "register">) { + let requested = 0, completed = -1; + const worker = coordinator.register(DELIVERY_QUEUES.runtimeAssetRetention, { + retryMs: 5_000, pausePassiveMaintenance: true, + async run(signal) { const generation = requested; await tick(signal); completed = generation; }, + hasPending: async () => false, + nextRunAt: () => requested !== completed ? Date.now() + 5_000 : next, + }); + const wake = () => { requested++; worker.wake(); }; + observers.add(wake); + return { ...worker, async stop() { observers.delete(wake); await worker.stop(); } }; + } + return { tick, start, nextRunAt: () => next }; +} diff --git a/server/src/services/native-runtime/runtime-context.test.ts b/server/src/services/native-runtime/runtime-context.test.ts index 5af85205e21..e6d1fbd9539 100644 --- a/server/src/services/native-runtime/runtime-context.test.ts +++ b/server/src/services/native-runtime/runtime-context.test.ts @@ -1,3 +1,4 @@ +import fs from "node:fs/promises"; import { mkdtemp, mkdir, readFile, readdir, chmod, lstat, rm, stat, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -290,7 +291,11 @@ describe("buildNativeRuntimeContext", () => { }; const context = await buildNativeRuntimeContext(input); + const reads = vi.spyOn(fs, "readFile"); const repeated = await buildNativeRuntimeContext(input); + expect(reads.mock.calls.some(([file]) => String(file).startsWith(selectedRoot) + || String(file).startsWith(context.skills[0]!.bundle.rootPath))).toBe(false); + reads.mockRestore(); expect(serviceMocks.exportFiles).toHaveBeenCalledWith(input.agent, { rejectSymlinks: true }); expect(serviceMocks.getEffectiveToolAssignmentsForAgent).toHaveBeenCalledWith("company-1", "agent-1"); @@ -322,6 +327,19 @@ describe("buildNativeRuntimeContext", () => { expect(repeated.aggregateDigest).toBe(context.aggregateDigest); expect(repeated.instructions.bundle.rootPath).toBe(context.instructions.bundle.rootPath); expect(repeated.skills[0]!.bundle.rootPath).toBe(context.skills[0]!.bundle.rootPath); + await writeFile(path.join(selectedRoot, "references/checklist.md"), "- Verify edits\n"); + const edited = await buildNativeRuntimeContext(input); + expect(edited.skills[0]!.bundle.digest).not.toBe(context.skills[0]!.bundle.digest); + const corrupted = path.join(edited.skills[0]!.bundle.rootPath, "SKILL.md"); + await chmod(corrupted, 0o644); + await writeFile(corrupted, "Corrupted saved asset"); + const repaired = await buildNativeRuntimeContext(input); + expect(repaired.skills[0]!.bundle.digest).toBe(edited.skills[0]!.bundle.digest); + expect(await readFile(corrupted, "utf8")).toContain("Use the checklist"); + await chmod(path.dirname(corrupted), 0o755); + await fs.rm(corrupted); + expect((await buildNativeRuntimeContext(input)).skills[0]!.bundle.digest).toBe(edited.skills[0]!.bundle.digest); + expect(await readFile(corrupted, "utf8")).toContain("Use the checklist"); }); it("does not materialize assigned skills or external MCP in preconfigured mode", async () => { diff --git a/server/src/services/native-runtime/runtime-context.ts b/server/src/services/native-runtime/runtime-context.ts index ee457686486..d0a96fed9de 100644 --- a/server/src/services/native-runtime/runtime-context.ts +++ b/server/src/services/native-runtime/runtime-context.ts @@ -1,3 +1,6 @@ +import { withRuntimeAssetUse, removeRuntimeAssetTree } from "./runtime-asset-retention.js"; +import { cachedSkillAsset, rememberSkillAsset } from "./skill-asset-cache.js"; +import { verifyFileManifest, type FileManifestEntry } from "../file-manifest-cache.js"; import { measureRunPhase } from "../execution-preparation-timing.js"; import { spekoToolsForSession } from "../voice/speko-agent-tools.js"; import { githubBotConnectionIdsForRun } from "../chat-github-tools.js"; @@ -83,14 +86,9 @@ async function verifyMaterializedAsset( rootPath: string, manifestFiles: Array<{ path: string; sha256: string; mode: number; size: number }>, ): Promise { - const actual = await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(rootPath)); - const actualByPath = new Map(actual.map((file) => [file.path, file])); - if (actualByPath.size !== manifestFiles.length) throw new Error("runtime context asset file count mismatch"); - for (const expected of manifestFiles) { - const file = actualByPath.get(expected.path); - if (!file || file.content.byteLength !== expected.size || sha256(file.content) !== expected.sha256 || (file.mode & 0o555) !== expected.mode) { - throw new Error(`runtime context asset digest mismatch: ${expected.path}`); - } + const expected = manifestFiles.map(file => ({ ...file, executable: Boolean(file.mode & 0o111) })); + if (!await measureRunPhase("runtime_context.asset_read", () => verifyFileManifest(rootPath, expected))) { + throw new Error("runtime context asset digest mismatch"); } } @@ -105,55 +103,62 @@ export async function materializeAsset(files: AssetFile[]): Promise fs.readFile(manifestPath, "utf8").catch(() => null)); - if (existing !== null) { - if (sha256(existing) !== manifestDigest || !(await fs.stat(rootPath).catch(() => null))?.isDirectory()) throw new Error(`runtime context asset verification failed: ${assetDigest}`); - await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes }); - return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; - } - await measureRunPhase("runtime_context.asset_materialize", async () => { - const stagingRoot = path.join(assetsRoot, ".staging", randomUUID()); - await fs.mkdir(stagingRoot, { recursive: true, mode: 0o700 }); - try { - for (const file of sorted) { - const target = path.join(stagingRoot, file.path); - await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); - await fs.writeFile(target, file.content, { mode: file.mode & 0o555 }); - await fs.chmod(target, file.mode & 0o555); - } - await fs.mkdir(path.dirname(rootPath), { recursive: true, mode: 0o700 }); - let publishedRoot = false; - await fs.rename(stagingRoot, rootPath).then(() => { - publishedRoot = true; - }).catch(async (error: NodeJS.ErrnoException) => { - // Concurrent publishers can surface EACCES on macOS when the winning - // destination has already been made read-only. Treat that as a race only - // when the content-addressed destination now exists; a genuine permission - // failure with no published destination must still fail loudly. - const publishedByPeer = (await fs.stat(rootPath).catch(() => null))?.isDirectory() === true; - if (error.code !== "EEXIST" && error.code !== "ENOTEMPTY" && !publishedByPeer) throw error; - await fs.rm(stagingRoot, { recursive: true, force: true }); - }); - // Publish the writable staging directory first. Renaming a read-only - // directory can fail with EACCES on macOS even when both parents are owned - // and writable. Files are already immutable; directories become immutable - // immediately after this process wins publication. - if (publishedRoot) await makeDirectoriesReadOnly(rootPath); - await fs.mkdir(path.dirname(manifestPath), { recursive: true, mode: 0o700 }); - await fs.writeFile(manifestPath, manifestText, { flag: "wx", mode: 0o400 }).catch(async (error: NodeJS.ErrnoException) => { - if (error.code !== "EEXIST") throw error; - if (sha256(await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8"))) !== manifestDigest) throw new Error(`runtime context manifest race mismatch: ${assetDigest}`); - }); - await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes }); - } catch (error) { - await fs.rm(stagingRoot, { recursive: true, force: true }).catch(() => undefined); - throw error; + return withRuntimeAssetUse(assetDigest, async function publishOrReuse(locked): Promise { + const assetsRoot = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets"); + const rootPath = path.join(assetsRoot, "bundles", assetDigest); + const manifestPath = path.join(assetsRoot, "manifests", `${assetDigest}.json`); + const existing = await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8").catch(() => null)); + const rootExists = (await fs.stat(rootPath).catch(() => null))?.isDirectory(); + if (existing !== null && rootExists && sha256(existing) === manifestDigest + && await verifyFileManifest(rootPath, manifestFiles.map(file => ({ ...file, executable: Boolean(file.mode & 0o111) })))) { + return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; } - }, { fileCount: manifestFiles.length, bytes: totalBytes }); - return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; + if (!locked) return withRuntimeAssetUse(assetDigest, publishOrReuse, true); + // The caller supplied the exact authoritative bytes for this digest. A missing + // or damaged disposable copy can be rebuilt under the publication/GC lock. + if (rootExists) await removeRuntimeAssetTree(rootPath); + await fs.rm(manifestPath, { force: true }); + await measureRunPhase("runtime_context.asset_materialize", async () => { + const stagingRoot = path.join(assetsRoot, ".staging", randomUUID()); + await fs.mkdir(stagingRoot, { recursive: true, mode: 0o700 }); + try { + for (const file of sorted) { + const target = path.join(stagingRoot, file.path); + await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); + await fs.writeFile(target, file.content, { mode: file.mode & 0o555 }); + await fs.chmod(target, file.mode & 0o555); + } + await fs.mkdir(path.dirname(rootPath), { recursive: true, mode: 0o700 }); + let publishedRoot = false; + await fs.rename(stagingRoot, rootPath).then(() => { + publishedRoot = true; + }).catch(async (error: NodeJS.ErrnoException) => { + // Concurrent publishers can surface EACCES on macOS when the winning + // destination has already been made read-only. Treat that as a race only + // when the content-addressed destination now exists; a genuine permission + // failure with no published destination must still fail loudly. + const publishedByPeer = (await fs.stat(rootPath).catch(() => null))?.isDirectory() === true; + if (error.code !== "EEXIST" && error.code !== "ENOTEMPTY" && !publishedByPeer) throw error; + await fs.rm(stagingRoot, { recursive: true, force: true }); + }); + // Publish the writable staging directory first. Renaming a read-only + // directory can fail with EACCES on macOS even when both parents are owned + // and writable. Files are already immutable; directories become immutable + // immediately after this process wins publication. + if (publishedRoot) await makeDirectoriesReadOnly(rootPath); + await fs.mkdir(path.dirname(manifestPath), { recursive: true, mode: 0o700 }); + await fs.writeFile(manifestPath, manifestText, { flag: "wx", mode: 0o400 }).catch(async (error: NodeJS.ErrnoException) => { + if (error.code !== "EEXIST") throw error; + if (sha256(await measureRunPhase("runtime_context.asset_cache", () => fs.readFile(manifestPath, "utf8"))) !== manifestDigest) throw new Error(`runtime context manifest race mismatch: ${assetDigest}`); + }); + await measureRunPhase("runtime_context.asset_verify", () => verifyMaterializedAsset(rootPath, manifestFiles), { fileCount: manifestFiles.length, bytes: totalBytes }); + } catch (error) { + await fs.rm(stagingRoot, { recursive: true, force: true }).catch(() => undefined); + throw error; + } + }, { fileCount: manifestFiles.length, bytes: totalBytes }); + return { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest: assetDigest, manifestDigest, rootPath, fileCount: manifestFiles.length, totalBytes }; + }); } async function materializeInstructionBundle(db: Db, agent: RuntimeAgent, agentFiles = false) { @@ -180,7 +185,7 @@ async function materializeInstructionBundle(db: Db, agent: RuntimeAgent, agentFi return { entryPath, bundle: await materializeAsset(files) }; } -async function materializeSelectedSkills(runtimeConfig: Record, entries: PaperclipSkillEntry[], omitLegacy: boolean) { +export async function materializeSelectedSkills(runtimeConfig: Record, entries: PaperclipSkillEntry[], omitLegacy: boolean) { const desiredKeys = resolvePaperclipDesiredSkillNames(runtimeConfig, entries).filter( (key) => !omitLegacy || key !== PAPERCLIP_OPERATIONAL_SKILL_KEY, ); @@ -189,7 +194,17 @@ async function materializeSelectedSkills(runtimeConfig: Record, const entry = byKey.get(key); if (!entry) throw new Error(`assigned runtime skill is missing from the company library: ${key}`); if (entry.sourceStatus === "missing") throw new Error(entry.missingDetail ?? `assigned runtime skill source is missing: ${key}`); - return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle: await materializeAsset(await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(entry.source))) }; + let bundle = await measureRunPhase("runtime_context.asset_verify", () => cachedSkillAsset(entry.source)); + if (!bundle) { + const files = await measureRunPhase("runtime_context.asset_read", () => collectDirectoryFiles(entry.source)); + bundle = await materializeAsset(files); + const manifest: FileManifestEntry[] = files.map(file => ({ path: file.path, size: file.content.length, + sha256: sha256(file.content), mode: file.mode & 0o555, executable: Boolean(file.mode & 0o111) })); + // Populate the stat/digest cache against these exact source bytes before + // remembering the reference. A concurrent source edit cannot seed it. + if (await verifyFileManifest(entry.source, manifest)) rememberSkillAsset(entry.source, manifest, bundle); + } + return { key: entry.key, runtimeName: safeRelativePath(entry.runtimeName, `runtime name for ${key}`), versionId: entry.versionId ?? entry.currentVersionId ?? null, bundle }; })); } diff --git a/server/src/services/native-runtime/skill-asset-cache.ts b/server/src/services/native-runtime/skill-asset-cache.ts new file mode 100644 index 00000000000..1a85783db5f --- /dev/null +++ b/server/src/services/native-runtime/skill-asset-cache.ts @@ -0,0 +1,46 @@ +import { withRuntimeAssetUse } from "./runtime-asset-retention.js"; +import fs from "node:fs/promises"; +import path from "node:path"; +import { createHash } from "node:crypto"; +import type { NativeRuntimeAssetReference } from "../../vendor/paperclip-runner/index.js"; +import { verifyFileManifest, type FileManifestEntry } from "../file-manifest-cache.js"; +import { resolvePaperclipInstanceRoot } from "../../home-paths.js"; + +type Cached = { manifest: FileManifestEntry[]; bundle: NativeRuntimeAssetReference }; +const entries = new Map(); +let fileCount = 0; +const MAX_FILES = 32_000; +const MAX_ENTRIES = 128; +const cacheKey = (source: string) => `${resolvePaperclipInstanceRoot()}:${source}`; + +/** The caller supplies freshly authorized runtime entries on every turn. This + * cache retains no authority or file bodies, and verifies source and destination + * identities before reusing the content-addressed native asset reference. */ +export async function cachedSkillAsset(source: string): Promise { + const cached = entries.get(cacheKey(source)); + if (!cached || !await verifyFileManifest(source, cached.manifest)) return null; + return withRuntimeAssetUse(cached.bundle.digest, async () => { + const manifestPath = path.join(resolvePaperclipInstanceRoot(), "runtime-context-assets", "manifests", `${cached.bundle.digest}.json`); + const manifest = await fs.readFile(manifestPath).catch(() => null); + if (!manifest || !(await fs.lstat(cached.bundle.rootPath).catch(() => null))?.isDirectory()) return null; + if (createHash("sha256").update(manifest).digest("hex") !== cached.bundle.manifestDigest + || !await verifyFileManifest(cached.bundle.rootPath, cached.manifest)) { + return null; + } + return cached.bundle; + }); +} + +export function rememberSkillAsset(source: string, manifest: FileManifestEntry[], bundle: NativeRuntimeAssetReference) { + if (manifest.length > MAX_FILES) return; + const key = cacheKey(source); + const previous = entries.get(key); + if (previous) { fileCount -= previous.manifest.length; entries.delete(key); } + while (entries.size && (fileCount + manifest.length > MAX_FILES || entries.size >= MAX_ENTRIES)) { + const first = entries.keys().next().value!; + fileCount -= entries.get(first)!.manifest.length; + entries.delete(first); + } + entries.set(key, { manifest, bundle }); + fileCount += manifest.length; +} diff --git a/server/src/services/skill-package-cache.ts b/server/src/services/skill-package-cache.ts new file mode 100644 index 00000000000..1966232cc49 --- /dev/null +++ b/server/src/services/skill-package-cache.ts @@ -0,0 +1,59 @@ +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { and, eq, inArray } from "drizzle-orm"; +import { companySkillRepositorySnapshots as snapshots, type Db } from "@paperclipai/db"; +import { ensureSkillCacheDirectory } from "./skill-version-cache.js"; + +const MAX_BYTES = 512 * 1024 * 1024; +const MAX_PACKAGES = 64; +const used = new Map(); +export function touchSkillPackage(directory: string) { + if (used.size >= 4096) used.delete(used.keys().next().value!); + used.set(directory, Date.now()); +} + +/** Called only after a cold fill, under the company cache lock. Never on a warm + * check. Version projections and run pins have their own existing retention. */ +export async function trimSkillPackageCache(db: Db, companyId: string, directory: string, + verifyDurable: (id: string, signal: AbortSignal) => Promise) { + const parent = path.dirname(directory); + await ensureSkillCacheDirectory(parent); + const contents = await fs.readdir(parent, { withFileTypes: true }); + for (const entry of contents) { + if (!/^\.(package|upload|verify)-[a-zA-Z0-9]{6}(\.tar\.gz)?$/.test(entry.name)) continue; + const target = path.join(parent, entry.name); + // Every I/O attempt has a three-minute deadline; leave a much longer crash + // grace so housekeeping cannot race a live publisher in another process. + if (Date.now() - (await fs.lstat(target)).mtimeMs > 60 * 60_000) await fs.rm(target, { recursive: true, force: true }); + } + const names = contents.filter(entry => entry.isDirectory() && /^[a-f0-9-]{36}$/i.test(entry.name)).map(entry => entry.name); + if (!names.length) return; + const rows = await db.select({ id: snapshots.id, storage: snapshots.storage }).from(snapshots) + .where(and(eq(snapshots.companyId, companyId), inArray(snapshots.id, names))); + const byId = new Map(rows.map(row => [row.id, row.storage])); + const entries = await Promise.all(names.map(async id => { + const location = path.join(parent, id); + const receipt = byId.get(id); + return { id, location, receipt, lastUsed: used.get(location) ?? (await fs.lstat(location)).mtimeMs, + // Include an upper bound on block rounding and directory/manifest overhead. + bytes: receipt ? receipt.totalBytes + receipt.fileCount * 8192 : 64 * 1024 * 1024 }; + })); + let bytes = entries.reduce((total, entry) => total + entry.bytes, 0); + let count = entries.length; + const signal = AbortSignal.timeout(10_000); + let attempted = 0; + for (const entry of entries.sort((a, b) => a.lastUsed - b.lastUsed)) { + if (bytes <= MAX_BYTES && count <= MAX_PACKAGES) break; + if (signal.aborted || attempted >= 2) break; + if (entry.location === directory) continue; + attempted++; + // A storage outage must not discard the last recoverable copy. Keep the + // excess until the next cold fill can confirm a durable copy independently. + if (byId.has(entry.id) && !await verifyDurable(entry.id, signal)) break; + await fs.rm(entry.location, { recursive: true, force: true }); + await fs.rm(path.join(parent, `${entry.id}.json`), { force: true }); + used.delete(entry.location); + bytes -= entry.bytes; + count--; + } +} diff --git a/server/src/services/skill-package-files.ts b/server/src/services/skill-package-files.ts new file mode 100644 index 00000000000..6ef087405a7 --- /dev/null +++ b/server/src/services/skill-package-files.ts @@ -0,0 +1,143 @@ +import { createHash } from "node:crypto"; +import { constants, createReadStream, createWriteStream, promises as fs } from "node:fs"; +import path from "node:path"; +import { Transform } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import { createGzip, createGunzip } from "node:zlib"; +import { extract, pack } from "tar-stream"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { assertSkillSnapshotPath, skillFileBytes, skillSnapshotHash } from "./skill-snapshot.js"; + +export const MAX_SKILL_PACKAGE_BYTES = 64 * 1024 * 1024; +export const MAX_SKILL_PACKAGE_FILES = 10_000; +export const MAX_SKILL_MANIFEST_BYTES = 48 * 1024 * 1024; +export type SkillPackageFile = { + path: string; + kind: CompanySkillVersionFileInventoryEntry["kind"]; + encoding: "utf8" | "base64"; + executable: boolean; + size: number; + sha256: string; +}; +export type SkillPackageManifest = { + format: "paperclip-skill-package/v1"; + contentHash: string; + totalBytes: number; + files: SkillPackageFile[]; +}; +const kinds = new Set(["skill", "markdown", "reference", "script", "asset", "other"]); +export const packageSha256 = (bytes: string | Buffer) => createHash("sha256").update(bytes).digest("hex"); + +export function validateSkillPackageManifest(value: unknown): SkillPackageManifest { + const manifest = value as SkillPackageManifest; + if (!manifest || manifest.format !== "paperclip-skill-package/v1" || !/^[a-f0-9]{64}$/.test(manifest.contentHash) + || !Array.isArray(manifest.files) || !manifest.files.length || manifest.files.length > MAX_SKILL_PACKAGE_FILES) { + throw new Error("Invalid skill package manifest"); + } + const paths = new Set(); + let total = 0; + for (const file of manifest.files) { + assertSkillSnapshotPath(file.path); + if (file.path.length > 4096 || file.path.split("/").length > 64 || paths.has(file.path) + || !kinds.has(file.kind) || !["utf8", "base64"].includes(file.encoding) || typeof file.executable !== "boolean" + || !Number.isSafeInteger(file.size) || file.size < 0 || !/^[a-f0-9]{64}$/.test(file.sha256)) throw new Error("Invalid skill package file"); + paths.add(file.path); + total += file.size; + } + for (const file of manifest.files) { + let parent = path.posix.dirname(file.path); + while (parent !== ".") { + if (paths.has(parent)) throw new Error("Conflicting skill package paths"); + parent = path.posix.dirname(parent); + } + } + if (total !== manifest.totalBytes || total > MAX_SKILL_PACKAGE_BYTES) throw new Error("Skill package exceeds its byte bound"); + return manifest; +} + +export function skillPackageManifest(files: CompanySkillVersionFileInventoryEntry[]): SkillPackageManifest { + return validateSkillPackageManifest({ + format: "paperclip-skill-package/v1", contentHash: skillSnapshotHash(files), + totalBytes: files.reduce((sum, file) => sum + skillFileBytes(file).length, 0), + files: files.map(file => { + const bytes = skillFileBytes(file); + return { path: file.path, kind: file.kind, encoding: file.encoding ?? "utf8", executable: Boolean(file.executable), size: bytes.length, sha256: packageSha256(bytes) }; + }), + }); +} + +/** JSONB reorders object keys. Preserve the archive manifest's v1 wire encoding + * when a recovery recipe is read back from PostgreSQL. Array order is retained. */ +export function skillPackageManifestBytes(manifest: SkillPackageManifest): Buffer { + return Buffer.from(JSON.stringify({ format: manifest.format, contentHash: manifest.contentHash, totalBytes: manifest.totalBytes, + files: manifest.files.map(file => ({ path: file.path, kind: file.kind, encoding: file.encoding, + executable: file.executable, size: file.size, sha256: file.sha256 })) })); +} + +export function boundedHashStream(maxBytes: number) { + const hash = createHash("sha256"); + let bytes = 0; + const stream = new Transform({ transform(chunk: Buffer, _encoding, done) { + bytes += chunk.length; + if (bytes > maxBytes) return done(new Error("Skill package stream exceeds its byte bound")); + hash.update(chunk); + done(null, chunk); + } }); + return { stream, result: () => ({ bytes, sha256: hash.digest("hex") }) }; +} + +export async function writeSkillPackageArchive(filename: string, files: CompanySkillVersionFileInventoryEntry[], signal?: AbortSignal) { + const archive = pack(); + const digest = boundedHashStream(MAX_SKILL_PACKAGE_BYTES + MAX_SKILL_MANIFEST_BYTES); + const writing = pipeline(archive, createGzip({ level: 3 }), digest.stream, createWriteStream(filename, { flags: "wx", mode: 0o600 }), { signal }); + // Attach a handler before producing entries so an early filesystem failure is observed. + void writing.catch(() => {}); + try { + for (const file of [...files].sort((a, b) => a.path.localeCompare(b.path))) { + signal?.throwIfAborted(); + await new Promise((resolve, reject) => archive.entry({ + name: `files/${assertSkillSnapshotPath(file.path)}`, type: "file", mode: file.executable ? 0o555 : 0o444, + mtime: new Date(0), uid: 0, gid: 0, + }, skillFileBytes(file), error => error ? reject(error) : resolve())); + } + archive.finalize(); + await writing; + return digest.result(); + } catch (error) { + archive.destroy(error as Error); + await writing.catch(() => {}); + throw error; + } +} + +export async function extractSkillPackageArchive(filename: string, directory: string, manifest: SkillPackageManifest, signal?: AbortSignal) { + const archive = extract(); + const expected = new Map(manifest.files.map(file => [`files/${file.path}`, file])); + const expanded = boundedHashStream(MAX_SKILL_PACKAGE_BYTES + MAX_SKILL_PACKAGE_FILES * 8192); + const reading = pipeline(createReadStream(filename), createGunzip(), expanded.stream, archive, { signal }); + void reading.catch(() => {}); + try { + for await (const entry of archive) { + signal?.throwIfAborted(); + const file = expected.get(entry.header.name); + if (!file || entry.header.type !== "file" || entry.header.size !== file.size + || entry.header.mode !== (file.executable ? 0o555 : 0o444)) throw new Error("Unexpected skill package archive entry"); + expected.delete(entry.header.name); + const target = path.join(directory, file.path); + await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); + const digest = boundedHashStream(file.size); + const output = await fs.open(target, constants.O_CREAT | constants.O_EXCL | constants.O_WRONLY | constants.O_NOFOLLOW, file.executable ? 0o555 : 0o444); + await pipeline(entry, digest.stream, output.createWriteStream(), { signal }); + const actual = digest.result(); + if (actual.bytes !== file.size || actual.sha256 !== file.sha256) throw new Error("Skill package file checksum mismatch"); + } + await reading; + if (expected.size) throw new Error("Skill package archive is incomplete"); + } catch (error) { + archive.destroy(error as Error); + await reading.catch(() => {}); + throw error; + } +} + +export { verifyFileManifest as verifySkillPackageDirectory, relocateFileManifestVerification as relocateSkillPackageVerification } from "./file-manifest-cache.js"; diff --git a/server/src/services/skill-package-maintenance.ts b/server/src/services/skill-package-maintenance.ts new file mode 100644 index 00000000000..d52315d72f7 --- /dev/null +++ b/server/src/services/skill-package-maintenance.ts @@ -0,0 +1,129 @@ +import path from "node:path"; +import { and, asc, eq, gt, isNull, lte, sql } from "drizzle-orm"; +import { companySkillRepositorySnapshots as snapshots, companySkillVersions as versions, + skillPackageDeletions, type Db } from "@paperclipai/db"; +import { getStorageService, type StorageService } from "../storage/index.js"; +import { skillPackageStore, skillVersionMetadataColumns } from "./skill-package-store.js"; +import { packageSha256 } from "./skill-package-files.js"; +import { skillFileBytes, skillSnapshotHash } from "./skill-snapshot.js"; + +type PageOptions = { id?: string; signal?: AbortSignal; companyId?: string; after?: string; limit?: number; apply?: boolean; clearLegacy?: boolean }; +const pageSize = (options: PageOptions) => Math.min(50, Math.max(1, options.limit ?? 10)); + +/** Bounded maintenance shared by automatic delivery and optional operator diagnostics. */ +export function skillPackageMaintenance(db: Db, storage?: StorageService) { + const store = skillPackageStore(db, storage); + async function migrateSnapshots(options: PageOptions = {}) { + const rows = await db.select({ id: snapshots.id, companyId: snapshots.companyId, contentHash: snapshots.contentHash }) + .from(snapshots).where(and(options.companyId ? eq(snapshots.companyId, options.companyId) : undefined, + options.id ? eq(snapshots.id, options.id) : undefined, + options.after ? gt(snapshots.id, options.after) : undefined, options.clearLegacy ? undefined : isNull(snapshots.storage))) + .orderBy(asc(snapshots.id)).limit(pageSize(options)); + let migrated = 0; + if (options.apply) for (const row of rows) { + // Transfer only one legacy body at a time. Never trust a mutable upstream. + const [legacy] = await db.select({ files: snapshots.fileInventory }).from(snapshots).where(eq(snapshots.id, row.id)); + options.signal?.throwIfAborted(); + if (!legacy?.files.length) { + await db.update(snapshots).set({ migrationAt: null }).where(eq(snapshots.id, row.id)); + continue; + } + if (skillSnapshotHash(legacy.files) !== row.contentHash) throw new Error(`Legacy package hash mismatch: ${row.id}`); + const id = await store.prepare(row.companyId, legacy.files, true, options.signal); + if (id !== row.id || !await store.verifyDurable(row.companyId, id, options.signal)) throw new Error(`Package verification failed: ${row.id}`); + // An ambiguous result is safe to retry: storage is already published, IDs + // are unchanged, and the old bytes survive until this final update. + if (options.clearLegacy) await db.update(snapshots).set({ fileInventory: [], migrationAt: null }).where(and(eq(snapshots.id, id), eq(snapshots.contentHash, row.contentHash), sql`${snapshots.storage} is not null`)); + migrated++; + } + return { inspected: rows.length, migrated, after: rows.at(-1)?.id ?? null }; + } + + async function migrateVersions(options: PageOptions = {}) { + const rows = await db.select(skillVersionMetadataColumns).from(versions).where(and( + options.companyId ? eq(versions.companyId, options.companyId) : undefined, + options.id ? eq(versions.id, options.id) : undefined, + options.after ? gt(versions.id, options.after) : undefined, options.clearLegacy ? undefined : isNull(versions.packageSnapshotId), isNull(versions.releaseId))) + .orderBy(asc(versions.id)).limit(pageSize(options)); + let migrated = 0; + if (options.apply) for (const row of rows) { + const [legacy] = await db.select({ files: versions.fileInventory }).from(versions).where(eq(versions.id, row.id)); + const files = legacy?.files; + options.signal?.throwIfAborted(); + if (!files?.length) { + await db.update(versions).set({ migrationAt: null }).where(eq(versions.id, row.id)); + continue; + } + if (row.repositorySnapshotId) await migrateSnapshots({ id: row.repositorySnapshotId, apply: true, clearLegacy: true, signal: options.signal }); + let packageId = row.packageSnapshotId; + let fileMap = row.packageFileMap; + if (packageId && skillSnapshotHash(await store.versionFiles(row)) !== skillSnapshotHash(files)) throw new Error(`Legacy version differs from its package: ${row.id}`); + if (!packageId && row.repositorySnapshotId && row.repositorySkillPath) { + const manifest = await store.manifest(row.companyId, row.repositorySnapshotId); + const byPath = new Map(manifest.files.map(file => [file.path, file])); + const mappings = files.map(file => ({ path: file.path, kind: file.kind, + sourcePath: file.path === "SKILL.md" ? row.repositorySkillPath! : path.posix.join(path.posix.dirname(row.repositorySkillPath!), file.path) })); + if (files.every((file, index) => { + const source = byPath.get(mappings[index]!.sourcePath); + return source?.sha256 === packageSha256(skillFileBytes(file)) && source.executable === Boolean(file.executable); + })) { packageId = row.repositorySnapshotId; fileMap = mappings; } + } + packageId ??= await store.prepare(row.companyId, files, true, options.signal); + const receipt = await store.descriptor(row.companyId, packageId); + if (!receipt.storage || !await store.verifyDurable(row.companyId, packageId, options.signal)) throw new Error(`Package must be migrated and verified first: ${packageId}`); + await db.update(versions).set({ packageSnapshotId: packageId, packageFileMap: fileMap, ...(options.clearLegacy ? { fileInventory: [], migrationAt: null } : {}) }) + .where(and(eq(versions.companyId, row.companyId), eq(versions.id, row.id), row.packageSnapshotId ? eq(versions.packageSnapshotId, row.packageSnapshotId) : isNull(versions.packageSnapshotId))); + migrated++; + } + return { inspected: rows.length, migrated, after: rows.at(-1)?.id ?? null }; + } + + async function collectOrphans(options: PageOptions = {}) { + const unused = sql`not exists (select 1 from ${versions} where ${versions.repositorySnapshotId} = ${snapshots.id} or ${versions.packageSnapshotId} = ${snapshots.id})`; + const rows = await db.select({ id: snapshots.id }).from(snapshots).where(and( + options.companyId ? eq(snapshots.companyId, options.companyId) : undefined, + options.id ? eq(snapshots.id, options.id) : undefined, + options.after ? gt(snapshots.id, options.after) : undefined, + sql`${snapshots.createdAt} < now() - interval '1 day'`, + sql`(${snapshots.upload} is null or (${snapshots.upload}->>'expiresAt')::timestamptz < now())`, unused, + )).orderBy(asc(snapshots.id)).limit(pageSize(options)); + let removed = 0; + if (options.apply) for (const row of rows) { + // FK locking wins against a concurrent publication. The trigger persists + // deletion receipts in the same transaction, including company cascades. + const deleted = await db.delete(snapshots).where(and(eq(snapshots.id, row.id), unused)).returning({ id: snapshots.id }); + removed += deleted.length; + } + return { inspected: rows.length, removed, after: rows.at(-1)?.id ?? null }; + } + + async function deleteObjects(options: PageOptions = {}) { + const backend = storage ?? getStorageService(); + const rows = await db.select().from(skillPackageDeletions).where(and( + options.companyId ? eq(skillPackageDeletions.companyId, options.companyId) : undefined, + options.id ? eq(skillPackageDeletions.id, options.id) : undefined, + options.after ? gt(skillPackageDeletions.id, options.after) : undefined, + lte(skillPackageDeletions.deleteAfter, new Date()), eq(skillPackageDeletions.provider, backend.provider), + )).orderBy(asc(skillPackageDeletions.id)).limit(pageSize(options)); + let removed = 0; + if (options.apply) for (const row of rows) { + options.signal?.throwIfAborted(); + const [referenced] = await db.select({ id: snapshots.id }).from(snapshots).where(and(eq(snapshots.companyId, row.companyId), + sql`(${snapshots.storage}->>'archiveKey' = ${row.archiveKey} or ${snapshots.upload}->>'archiveKey' = ${row.archiveKey})`)).limit(1); + if (referenced) { + // Live ownership supersedes this receipt. A future replacement/deletion + // enqueues fresh work in the same transaction as the ownership change. + await db.delete(skillPackageDeletions).where(eq(skillPackageDeletions.id, row.id)); + continue; + } + for (const key of [row.archiveKey, row.manifestKey]) { + await backend.deleteObject(row.companyId, key); + if ((await backend.headObject(row.companyId, key)).exists) throw new Error(`Storage did not confirm package deletion: ${row.id}`); + } + await db.delete(skillPackageDeletions).where(eq(skillPackageDeletions.id, row.id)); + removed++; + } + return { inspected: rows.length, removed, after: rows.at(-1)?.id ?? null }; + } + return { migrateSnapshots, migrateVersions, collectOrphans, deleteObjects }; +} diff --git a/server/src/services/skill-package-recovery.test.ts b/server/src/services/skill-package-recovery.test.ts new file mode 100644 index 00000000000..cd338752237 --- /dev/null +++ b/server/src/services/skill-package-recovery.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it, vi } from "vitest"; +import { randomUUID } from "node:crypto"; +import { recoverSkillPackage } from "./skill-package-recovery.js"; +import { skillPackageManifest } from "./skill-package-files.js"; +import type { GitHubRead } from "./github-skill-source.js"; +import type { SkillPackageRestoreRecipe } from "@paperclipai/db"; + +function fixture() { + const files = [{ path: "SKILL.md", kind: "skill" as const, content: "---\nname: original\n---\nold commit" }, + { path: "run.sh", kind: "script" as const, content: "echo original", executable: true }]; + const recipe: SkillPackageRestoreRecipe = { version: 1, sourceId: randomUUID(), repositoryId: "123", repositoryUrl: "https://github.com/acme/skills", + commitSha: "a".repeat(40), skillPath: "skills/one/skill.md", manifest: skillPackageManifest(files) }; + const entries = recipe.manifest.files.map((file, i) => ({ path: `skills/one/${i ? "run.sh" : "skill.md"}`, type: "blob", mode: file.executable ? "100755" : "100644", sha: String(i).repeat(40), size: file.size })); + const release = vi.fn(async () => {}); + const readBlob = vi.fn(async (entry: typeof entries[number]) => Buffer.from(files[entries.indexOf(entry)]!.content)); + const read = Object.assign(vi.fn(async () => ({ id: 123 })), { + openSnapshot: vi.fn(async () => ({ commitSha: recipe.commitSha, entries, readBlob, release })), + }) as unknown as GitHubRead; + return { files, recipe, read, entries, readBlob, release }; +} +describe("immutable skill package recovery", () => { + it("reads the original manifest at the exact commit, including normalized entrypoint and modes", async () => { + const f = fixture(); + expect(await recoverSkillPackage(f.recipe, f.read, new AbortController().signal)).toEqual(f.files); + expect(f.read.openSnapshot).toHaveBeenCalledWith({ repositoryUrl: f.recipe.repositoryUrl, ref: f.recipe.commitSha, commitSha: f.recipe.commitSha }, expect.anything()); + expect(f.read).toHaveBeenCalledExactlyOnceWith("/repos/acme/skills", expect.any(AbortSignal)); + expect(f.release).toHaveBeenCalledOnce(); + }); + it("rejects reused repository names, wrong commits, changed contents and missing files", async () => { + const f = fixture(); + vi.mocked(f.read).mockResolvedValueOnce({ id: 999 }); + await expect(recoverSkillPackage(f.recipe, f.read, new AbortController().signal)).rejects.toThrow("not the original"); + expect(f.read.openSnapshot).not.toHaveBeenCalled(); + f.readBlob.mockResolvedValueOnce(Buffer.from("wrong")); + await expect(recoverSkillPackage(f.recipe, f.read, new AbortController().signal)).rejects.toThrow("checksum mismatch"); + f.entries.pop(); + await expect(recoverSkillPackage(f.recipe, f.read, new AbortController().signal)).rejects.toThrow("unavailable"); + expect(f.release).toHaveBeenCalledTimes(2); + }); +}); diff --git a/server/src/services/skill-package-recovery.ts b/server/src/services/skill-package-recovery.ts new file mode 100644 index 00000000000..7a56c94e72a --- /dev/null +++ b/server/src/services/skill-package-recovery.ts @@ -0,0 +1,62 @@ +import path from "node:path"; +import { and, eq } from "drizzle-orm"; +import { companySkillSources, type Db, type SkillPackageRestoreRecipe } from "@paperclipai/db"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { forbidden } from "../errors.js"; +import { assertSkillSnapshotPath } from "./skill-snapshot.js"; +import { packageSha256, skillPackageManifest, validateSkillPackageManifest } from "./skill-package-files.js"; +import type { GitHubRead } from "./github-skill-source.js"; + +/** Recheck the source's saved identity and current grant. Never search another + * company's credentials or turn a mutable branch name into a historical commit. */ +export async function skillPackageRecoveryReader(db: Db, companyId: string, recipe: SkillPackageRestoreRecipe) { + const [source] = await db.select().from(companySkillSources).where(and( + eq(companySkillSources.companyId, companyId), eq(companySkillSources.id, recipe.sourceId))); + if (!source || source.removedAt || !source.enabled || source.repositoryId !== recipe.repositoryId + || source.repositoryUrl.toLowerCase() !== recipe.repositoryUrl.toLowerCase()) { + throw forbidden("This skill package is no longer cached. Reconnect its original repository to restore it."); + } + // Deferred import avoids making the package store own source synchronization. + const { skillSourceSetupContext } = await import("./skill-source-sync.js"); + const context = await skillSourceSetupContext(db, source); + await context.authorize("skills.update", { sourceType: "git", sourceLocator: recipe.repositoryUrl }); + return context.read(source.connectionId); +} + +export async function recoverSkillPackage(recipe: SkillPackageRestoreRecipe, read: GitHubRead, signal: AbortSignal): Promise { + if (recipe.version !== 1 || !/^[a-f0-9]{40}$/i.test(recipe.commitSha)) throw new Error("Invalid immutable skill package recipe"); + const manifest = validateSkillPackageManifest(recipe.manifest); + const url = new URL(recipe.repositoryUrl); + if (url.protocol !== "https:" || url.hostname !== "github.com" || url.username || url.password + || !/^\/[^/]+\/[^/]+$/.test(url.pathname) || url.search || url.hash) throw new Error("Invalid skill repository identity"); + const repository = await read(`/repos${url.pathname}`, signal) as { id?: number | string }; + if (String(repository.id) !== recipe.repositoryId) throw new Error("The repository at this URL is not the original skill repository"); + const snapshot = await read.openSnapshot({ repositoryUrl: recipe.repositoryUrl, ref: recipe.commitSha, commitSha: recipe.commitSha }, { signal }); + try { + if (snapshot.commitSha !== recipe.commitSha) throw new Error("Git returned a different skill commit"); + const byPath = new Map(snapshot.entries.map(entry => [entry.path, entry])); + if (recipe.skillPath) assertSkillSnapshotPath(recipe.skillPath); + const entries = manifest.files.map(file => { + const sourcePath = recipe.skillPath + ? file.path === "SKILL.md" ? recipe.skillPath : path.posix.join(path.posix.dirname(recipe.skillPath), file.path) + : file.path; + const entry = byPath.get(assertSkillSnapshotPath(sourcePath)); + if (!entry || entry.type !== "blob" || entry.mode !== (file.executable ? "100755" : "100644") || entry.size !== file.size) { + throw new Error(`The original skill file is unavailable: ${sourcePath}`); + } + return entry; + }); + const blobs = snapshot.readBlobs ? await snapshot.readBlobs(entries, signal) : null; + const files: CompanySkillVersionFileInventoryEntry[] = []; + for (const [index, entry] of entries.entries()) { + signal.throwIfAborted(); + const expected = manifest.files[index]!; + const bytes = blobs?.get(entry.sha) ?? await snapshot.readBlob(entry, signal); + if (bytes.length !== expected.size || packageSha256(bytes) !== expected.sha256) throw new Error(`Restored skill file checksum mismatch: ${expected.path}`); + files.push({ path: expected.path, kind: expected.kind, content: bytes.toString(expected.encoding), + ...(expected.encoding === "base64" ? { encoding: expected.encoding } : {}), ...(expected.executable ? { executable: true } : {}) }); + } + if (skillPackageManifest(files).contentHash !== manifest.contentHash) throw new Error("Restored skill package checksum mismatch"); + return files; + } finally { await snapshot.release(); } +} diff --git a/server/src/services/skill-package-runtime.ts b/server/src/services/skill-package-runtime.ts new file mode 100644 index 00000000000..6e29f495370 --- /dev/null +++ b/server/src/services/skill-package-runtime.ts @@ -0,0 +1,120 @@ +import { measureRunPhase } from "./execution-preparation-timing.js"; +import { promises as fs } from "node:fs"; +import path from "node:path"; +import type { Db } from "@paperclipai/db"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { skillPackageStore, type StoredSkillVersion } from "./skill-package-store.js"; +import { packageSha256, verifySkillPackageDirectory, relocateSkillPackageVerification, type SkillPackageFile, type SkillPackageManifest } from "./skill-package-files.js"; +import { REPOSITORY_CONTENT_DIR, repositorySkillProjection } from "./skill-repository-package.js"; +import { assertSkillSnapshotPath, skillFileBytes } from "./skill-snapshot.js"; +import { ensureSkillCacheDirectory } from "./skill-version-cache.js"; + +type ProjectedFile = SkillPackageFile & { sourcePath?: string; generated?: Buffer }; +// Only the wrapper/policy bytes and file metadata are retained, never a decoded repository. +const projections = new Map(); +let projectionBytes = 0; +const MAX_PROJECTION_BYTES = 16 * 1024 * 1024; +const MAX_PROJECTIONS = 256; +const repositoryFiles = new WeakMap(); +function originalFiles(version: StoredSkillVersion, manifest: SkillPackageManifest): ProjectedFile[] { + if (version.repositorySnapshotId) { + let shared = repositoryFiles.get(manifest); + if (!shared) { + shared = manifest.files.map(file => ({ ...file, sourcePath: file.path, path: `${REPOSITORY_CONTENT_DIR}/${file.path}` })); + repositoryFiles.set(manifest, shared); + } + return [...shared]; + } + if (!version.packageFileMap) return manifest.files.map(file => ({ ...file, sourcePath: file.path })); + const byPath = new Map(manifest.files.map(file => [file.path, file])); + return version.packageFileMap.map(mapping => { + assertSkillSnapshotPath(mapping.path); + assertSkillSnapshotPath(mapping.sourcePath); + const file = byPath.get(mapping.sourcePath); + if (!file) throw new Error("Skill version references a missing package file"); + return { ...file, ...mapping }; + }); +} +export function skillPackageRuntime(db: Db) { + const store = skillPackageStore(db); + async function inventory(version: StoredSkillVersion) { + const id = version.repositorySnapshotId ?? version.packageSnapshotId; + if (!id) throw new Error("Skill version has no package"); + const manifest = await store.manifest(version.companyId, id); + const files = originalFiles(version, manifest).map(({ path, kind }) => ({ path, kind })); + if (version.repositorySnapshotId) { + files.unshift({ path: "SKILL.md", kind: "skill" }); + const policy = path.posix.join(path.posix.dirname(version.repositorySkillPath!), "agents/openai.yaml"); + const file = manifest.files.find(file => file.path === policy); + if (file) files.push({ path: "agents/openai.yaml", kind: file.kind }); + } + return files; + } + async function projection(version: StoredSkillVersion): Promise { + const id = version.repositorySnapshotId ?? version.packageSnapshotId; + if (!id) throw new Error("Skill version has no package"); + const manifest = await store.manifest(version.companyId, id); + const key = `${version.companyId}:${version.id}:${manifest.contentHash}`; + const files = originalFiles(version, manifest); + const cached = projections.get(key); + if (cached) return [...files, ...cached.files]; + const generatedFiles: ProjectedFile[] = []; + if (version.repositorySnapshotId) { + if (!version.repositorySkillPath) throw new Error("Repository skill entrypoint is missing"); + const generated = await store.withPackage(version.companyId, id, async (root, manifest) => { + const wanted = new Set([version.repositorySkillPath!, path.posix.join(path.posix.dirname(version.repositorySkillPath!), "agents/openai.yaml")]); + const selected: CompanySkillVersionFileInventoryEntry[] = []; + for (const file of manifest.files.filter(file => wanted.has(file.path))) selected.push({ + ...file, content: (await fs.readFile(path.join(root, file.path))).toString(file.encoding), + }); + return repositorySkillProjection(selected, version.repositorySkillPath!).filter(file => !file.path.startsWith(`${REPOSITORY_CONTENT_DIR}/`)); + }); + for (const file of generated) { + const bytes = skillFileBytes(file); + generatedFiles.push({ path: file.path, kind: file.kind, encoding: file.encoding ?? "utf8", executable: Boolean(file.executable), + size: bytes.length, sha256: packageSha256(bytes), generated: bytes }); + } + } + // Retain only tiny generated wrappers/policies per version. Shared package + // file metadata follows the bounded manifest cache through the WeakMap. + const bytes = generatedFiles.reduce((sum, file) => sum + (file.generated?.length ?? 0) + 512, 0); + while (projections.size && (projections.size >= MAX_PROJECTIONS || projectionBytes + bytes > MAX_PROJECTION_BYTES)) { + const first = projections.keys().next().value!; + projectionBytes -= projections.get(first)!.bytes; + projections.delete(first); + } + if (generatedFiles.length && bytes <= MAX_PROJECTION_BYTES) { projections.set(key, { files: generatedFiles, bytes }); projectionBytes += bytes; } + return [...files, ...generatedFiles]; + } + async function materialize(version: StoredSkillVersion, directory: string) { + const files = await projection(version); + const parent = path.dirname(directory); + await ensureSkillCacheDirectory(parent, true); + if (await measureRunPhase("skills.version.verify", () => verifySkillPackageDirectory(directory, files), { fileCount: files.length })) return directory; + return store.withPackage(version.companyId, (version.repositorySnapshotId ?? version.packageSnapshotId)!, async root => { + const staging = await fs.mkdtemp(path.join(parent, `.${version.id}-`)); + try { + for (const file of files) { + const target = path.join(staging, file.path); + await fs.mkdir(path.dirname(target), { recursive: true }); + if (file.generated) await fs.writeFile(target, file.generated); + else await fs.copyFile(path.join(root, file.sourcePath!), target); + await fs.chmod(target, file.executable ? 0o555 : 0o444); + } + if (!await verifySkillPackageDirectory(staging, files)) throw new Error("Prepared skill version failed integrity verification"); + await fs.rm(directory, { recursive: true, force: true }); + await fs.rename(staging, directory); + relocateSkillPackageVerification(staging, directory, files); + return directory; + } finally { await fs.rm(staging, { recursive: true, force: true }); } + }); + } + async function readFile(version: StoredSkillVersion, relative: string) { + const file = (await projection(version)).find(file => file.path === relative); + if (!file) return null; + const bytes = file.generated ?? await store.withPackage(version.companyId, + (version.repositorySnapshotId ?? version.packageSnapshotId)!, root => fs.readFile(path.join(root, file.sourcePath!))); + return { path: file.path, kind: file.kind, content: bytes.toString(file.encoding), encoding: file.encoding, executable: file.executable }; + } + return { inventory, materialize, readFile }; +} diff --git a/server/src/services/skill-package-store.ts b/server/src/services/skill-package-store.ts new file mode 100644 index 00000000000..b6df3b0056f --- /dev/null +++ b/server/src/services/skill-package-store.ts @@ -0,0 +1,443 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; +import { recoverSkillPackage, skillPackageRecoveryReader } from "./skill-package-recovery.js"; +import { touchSkillPackage, trimSkillPackageCache } from "./skill-package-cache.js"; +import { AsyncLocalStorage } from "node:async_hooks"; +import { randomUUID } from "node:crypto"; +import { constants, createReadStream, createWriteStream, promises as fs } from "node:fs"; +import path from "node:path"; +import { pipeline } from "node:stream/promises"; +import { and, eq, getTableColumns, inArray, sql } from "drizzle-orm"; +import { companySkillRepositorySnapshots as snapshots, companySkillVersions, type Db, type SkillPackageStorageReceipt, type SkillPackageRestoreRecipe } from "@paperclipai/db"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { getStorageService, type StorageService } from "../storage/index.js"; +import { resolvePaperclipInstanceRoot } from "../home-paths.js"; +import { conflict, notFound } from "../errors.js"; +import { assertSkillSnapshotPath, skillFileBytes } from "./skill-snapshot.js"; +import { ensureSkillCacheDirectory, withSkillVersionCacheLock } from "./skill-version-cache.js"; +import { boundedHashStream, extractSkillPackageArchive, MAX_SKILL_MANIFEST_BYTES, packageSha256, + skillPackageManifest, skillPackageManifestBytes, validateSkillPackageManifest, verifySkillPackageDirectory, relocateSkillPackageVerification, writeSkillPackageArchive, + type SkillPackageManifest } from "./skill-package-files.js"; + +/** External storage is the default; false is an explicit rolling-upgrade escape hatch. */ +export function skillPackageStorageEnabled() { + return process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED !== "false"; +} + +const descriptorColumns = { id: snapshots.id, companyId: snapshots.companyId, contentHash: snapshots.contentHash, + storage: snapshots.storage, upload: snapshots.upload, createdAt: snapshots.createdAt, + restorable: sql`${snapshots.restore} is not null`, restoreRetryAt: snapshots.restoreRetryAt }; +type Descriptor = Pick & { restorable: boolean }; +export const skillVersionMetadataColumns = { ...getTableColumns(companySkillVersions), + fileInventory: sql`'[]'::jsonb` }; +export type StoredSkillVersion = typeof companySkillVersions.$inferSelect; +type ReadScope = { db: Db; descriptors: Map>; durable: Map>; versions: Map }; +const readScope = new AsyncLocalStorage(); +export function withSkillPackageReadScope(db: Db, action: () => Promise): Promise { + return readScope.getStore()?.db === db ? action() : readScope.run({ db, descriptors: new Map(), durable: new Map(), versions: new Map() }, action); +} +const flights = new WeakMap>>(); +function flight(db: Db, key: string, action: () => Promise): Promise { + let pending = flights.get(db); + if (!pending) { pending = new Map(); flights.set(db, pending); } + const existing = pending.get(key); + if (existing) return existing as Promise; + const work = action().finally(() => pending!.delete(key)); + pending.set(key, work); + return work; +} +let activeIo = 0; +const waitingIo: Array<() => void> = []; +async function withPackageIo(action: () => Promise, signal: AbortSignal): Promise { + signal.throwIfAborted(); + if (activeIo >= 2) { + if (waitingIo.length >= 32) throw conflict("Skill package storage is busy; retry shortly."); + await new Promise((resolve, reject) => { + const ready = () => { signal.removeEventListener("abort", aborted); resolve(); }; + const aborted = () => { + const index = waitingIo.indexOf(ready); + if (index >= 0) waitingIo.splice(index, 1); + reject(signal.reason); + }; + waitingIo.push(ready); + signal.addEventListener("abort", aborted, { once: true }); + }); + } else activeIo++; + try { signal.throwIfAborted(); return await action(); } + finally { const next = waitingIo.shift(); if (next) next(); else activeIo--; } +} +const manifests = new Map(); +let manifestBytes = 0; +const MAX_MANIFEST_CACHE = 16 * 1024 * 1024; +function rememberManifest(key: string, manifest: SkillPackageManifest, bytes: number) { + if (bytes > MAX_MANIFEST_CACHE) return; + const previous = manifests.get(key); + if (previous) manifestBytes -= previous.bytes; + manifests.delete(key); + while (manifests.size && manifestBytes + bytes > MAX_MANIFEST_CACHE) { + const first = manifests.keys().next().value!; + manifestBytes -= manifests.get(first)!.bytes; + manifests.delete(first); + } + manifests.set(key, { manifest, bytes }); + manifestBytes += bytes; +} +export function skillPackagePaths(companyId: string, snapshotId: string) { + if (![companyId, snapshotId].every(id => /^[a-f0-9-]{36}$/i.test(id))) throw new Error("Invalid skill package identity"); + const parent = path.join(resolvePaperclipInstanceRoot(), "skills", companyId, "__packages__"); + return { parent, directory: path.join(parent, snapshotId), manifest: path.join(parent, `${snapshotId}.json`) }; +} +async function readBoundedObject(storage: StorageService, companyId: string, key: string, length: number, hash: string, signal: AbortSignal) { + if (!Number.isSafeInteger(length) || length < 0 || length > MAX_SKILL_MANIFEST_BYTES) throw new Error("Invalid skill manifest size"); + const object = await storage.getObject(companyId, key, { signal }); + const chunks: Buffer[] = []; + let bytes = 0; + try { + for await (const chunk of object.stream) { + const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); + bytes += buffer.length; + if (bytes > length) throw new Error("Skill manifest exceeds its recorded size"); + chunks.push(buffer); + } + } finally { object.stream.destroy(); } + const result = Buffer.concat(chunks); + if (bytes !== length || packageSha256(result) !== hash) throw new Error("Skill manifest checksum mismatch"); + return result; +} + +/** Owns immutable package storage. Authorization belongs to the referencing + * skill/source, and must be checked even when package bytes are already cached. */ +export function skillPackageStore(db: Db, suppliedStorage?: StorageService) { + const storage = () => suppliedStorage ?? getStorageService(); + const scope = (companyId: string, id: string) => and(eq(snapshots.companyId, companyId), eq(snapshots.id, id)); + async function descriptor(companyId: string, id: string): Promise { + const load = async () => { + const [row] = await db.select(descriptorColumns).from(snapshots).where(scope(companyId, id)); + if (!row || (!row.storage && !row.restorable && row.upload && !row.upload.legacy)) throw notFound("Installed skill package is unavailable"); + return row; + }; + const context = readScope.getStore(); + if (context?.db !== db) return load(); + const key = `${companyId}:${id}`; + if (!context.descriptors.has(key)) context.descriptors.set(key, load()); + return context.descriptors.get(key)!; + } + async function version(companyId: string, skillId: string, id: string): Promise { + const context = readScope.getStore(); + const cached = context?.db === db ? context.versions.get(`${companyId}:${id}`) : null; + if (cached) return cached.companySkillId === skillId ? cached : null; + const [row] = await db.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.companySkillId, skillId), eq(companySkillVersions.id, id))); + return row ?? null; + } + async function prefetch(companyId: string, versionIds: string[]) { + const context = readScope.getStore(); + if (context?.db !== db || versionIds.length < 2) return; + const rows = await db.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, companyId), inArray(companySkillVersions.id, [...new Set(versionIds)]))); + for (const row of rows) context.versions.set(`${companyId}:${row.id}`, row); + const ids = [...new Set(rows.flatMap(row => [row.repositorySnapshotId, row.packageSnapshotId]).filter((id): id is string => Boolean(id)))]; + if (!ids.length) return; + const packages = await db.select(descriptorColumns).from(snapshots).where(and(eq(snapshots.companyId, companyId), inArray(snapshots.id, ids))); + for (const row of packages) if (row.storage || row.restorable || !row.upload || row.upload.legacy) { + context.descriptors.set(`${companyId}:${row.id}`, Promise.resolve(row)); + } + } + async function restoreRecipe(row: Descriptor) { + const [saved] = await db.select({ restore: snapshots.restore }).from(snapshots).where(scope(row.companyId, row.id)); + const recipe = saved?.restore; + if (!recipe || recipe.version !== 1) throw notFound("This skill package has no exact-commit restore recipe"); + const manifest = validateSkillPackageManifest(recipe.manifest); + if (manifest.contentHash !== row.contentHash || (row.storage && packageSha256(skillPackageManifestBytes(manifest)) !== row.storage.manifestSha256)) { + throw new Error("Skill package restore recipe does not match its receipt"); + } + return recipe; + } + async function acquireFiles(row: Descriptor, signal: AbortSignal) { + signal.throwIfAborted(); + const [current] = await db.select({ retryAt: snapshots.restoreRetryAt }).from(snapshots).where(scope(row.companyId, row.id)); + if (current?.retryAt && current.retryAt.getTime() > Date.now()) throw conflict("The original skill package could not be acquired. Reconnect its repository and retry after the recovery backoff."); + try { + const recipe = await restoreRecipe(row); + const read = await skillPackageRecoveryReader(db, row.companyId, recipe); + const files = await recoverSkillPackage(recipe, read, signal); + await db.update(snapshots).set({ restoreRetryAt: null }).where(scope(row.companyId, row.id)); + return files; + } catch (error) { + if (!signal.aborted) await db.update(snapshots).set({ restoreRetryAt: new Date(Date.now() + 5 * 60_000) }).where(scope(row.companyId, row.id)); + throw error; + } + } + async function legacyFiles(row: Descriptor) { + const [legacy] = await db.select({ files: snapshots.fileInventory, storage: snapshots.storage, upload: snapshots.upload }) + .from(snapshots).where(scope(row.companyId, row.id)); + if (!legacy || (!legacy.files.length && !legacy.storage)) throw notFound("Installed skill package has no saved contents"); + // A reader may have loaded its descriptor just before a migration clears + // the legacy payload. Upgrade this request's descriptor to the published + // receipt instead of reporting a missing historical version. + if (legacy.storage) { row.storage = legacy.storage; row.upload = legacy.upload; } + return legacy.files; + } + async function manifestFor(row: Descriptor): Promise { + const paths = skillPackagePaths(row.companyId, row.id); + const key = `${paths.manifest}:${row.storage?.manifestSha256 ?? row.contentHash}`; + const cached = manifests.get(key); + if (cached) return cached.manifest; + return flight(db, `manifest:${key}`, async () => { + await ensureSkillCacheDirectory(paths.parent, true); + // Legacy manifests have no independently stored checksum. Derive them once + // per process from the authoritative row until the operator migrates it. + let bytes: Buffer | null = null; + if (row.storage) { + try { + const file = await fs.open(paths.manifest, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await file.stat(); + if (stat.isFile() && stat.size <= MAX_SKILL_MANIFEST_BYTES) bytes = await file.readFile(); + } finally { await file.close(); } + } catch (error) { + if (!["ENOENT", "ELOOP", "EISDIR"].includes((error as NodeJS.ErrnoException).code ?? "")) throw error; + } + } + if (row.storage) { + if (!bytes || packageSha256(bytes) !== row.storage.manifestSha256) { + const signal = AbortSignal.timeout(180_000); + try { + if (row.storage.provider !== storage().provider) throw new Error("Skill package storage provider changed"); + bytes = await withPackageIo(() => readBoundedObject(storage(), row.companyId, row.storage!.manifestKey, + row.storage!.manifestBytes, row.storage!.manifestSha256, signal), signal); + } catch (error) { + const [legacy] = await db.select({ files: snapshots.fileInventory }).from(snapshots).where(scope(row.companyId, row.id)); + const saved = legacy?.files.length ? skillPackageManifestBytes(skillPackageManifest(legacy.files)) : null; + if (saved && packageSha256(saved) === row.storage!.manifestSha256) bytes = saved; + else if (row.restorable) bytes = skillPackageManifestBytes((await restoreRecipe(row)).manifest); + else throw error; + } + } + } else if (row.restorable) { + bytes = skillPackageManifestBytes((await restoreRecipe(row)).manifest); + } else if (!bytes) { + const files = await legacyFiles(row); + if (row.storage) return manifestFor(row); + bytes = skillPackageManifestBytes(skillPackageManifest(files)); + } + if (bytes.length > MAX_SKILL_MANIFEST_BYTES) throw new Error("Skill manifest exceeds its byte bound"); + const manifest = validateSkillPackageManifest(JSON.parse(bytes.toString("utf8"))); + if (row.storage && (manifest.contentHash !== row.contentHash || manifest.files.length !== row.storage.fileCount + || manifest.totalBytes !== row.storage.totalBytes)) throw new Error("Skill package receipt does not match its manifest"); + await ensureSkillCacheDirectory(paths.parent, true); + const temp = `${paths.manifest}.${randomUUID()}`; + try { await fs.writeFile(temp, bytes, { flag: "wx", mode: 0o600 }); await fs.rename(temp, paths.manifest); } + finally { await fs.rm(temp, { force: true }); } + rememberManifest(key, manifest, bytes.length); + return manifest; + }); + } + async function materialize(row: Descriptor, manifest: SkillPackageManifest) { + const paths = skillPackagePaths(row.companyId, row.id); + return flight(db, `files:${paths.directory}`, async () => { + await ensureSkillCacheDirectory(paths.parent, true); + if (await verifySkillPackageDirectory(paths.directory, manifest.files)) return { directory: paths.directory, created: false }; + const signal = AbortSignal.timeout(180_000); + return withPackageIo(async () => { + const staging = await fs.mkdtemp(path.join(paths.parent, ".package-")); + const archivePath = `${staging}.tar.gz`; + try { + let legacy: CompanySkillVersionFileInventoryEntry[] | null = null; + if (!row.storage && !row.restorable) legacy = await legacyFiles(row); + try { + if (row.storage) { + if (row.storage.provider !== storage().provider) throw new Error("Skill package storage provider changed"); + const object = await storage().getObject(row.companyId, row.storage.archiveKey, { signal }); + const digest = boundedHashStream(row.storage.archiveBytes); + await pipeline(object.stream, digest.stream, createWriteStream(archivePath, { flags: "wx", mode: 0o600 }), { signal }); + const actual = digest.result(); + if (actual.bytes !== row.storage.archiveBytes || actual.sha256 !== row.storage.archiveSha256) throw new Error("Skill package archive checksum mismatch"); + await extractSkillPackageArchive(archivePath, staging, manifest, signal); + } + } catch (error) { + signal.throwIfAborted(); + // A missing/corrupt archive must not hide a still-authoritative legacy + // body. Only then use the saved, authorized immutable Git recipe. + const [saved] = await db.select({ files: snapshots.fileInventory }).from(snapshots).where(scope(row.companyId, row.id)); + if (saved?.files.length && skillPackageManifest(saved.files).contentHash === row.contentHash) legacy = saved.files; + else if (row.restorable) legacy = await acquireFiles(row, signal); + else throw error; + await fs.rm(staging, { recursive: true, force: true }); + await fs.mkdir(staging); + } + if (!row.storage && row.restorable && !legacy) legacy = await acquireFiles(row, signal); + if (legacy) { + for (const file of legacy) { + signal.throwIfAborted(); + const target = path.join(staging, assertSkillSnapshotPath(file.path)); + await fs.mkdir(path.dirname(target), { recursive: true, mode: 0o700 }); + await fs.writeFile(target, skillFileBytes(file), { flag: "wx", mode: file.executable ? 0o555 : 0o444 }); + } + } + if (!await verifySkillPackageDirectory(staging, manifest.files)) throw new Error("Restored skill package does not match its manifest"); + // All consumers hold the company cache lock, including processes sharing a volume. + await fs.rm(paths.directory, { recursive: true, force: true }); + await fs.rename(staging, paths.directory); + relocateSkillPackageVerification(staging, paths.directory, manifest.files); + return { directory: paths.directory, created: true }; + } finally { await fs.rm(staging, { recursive: true, force: true }); await fs.rm(archivePath, { force: true }); } + }, signal); + }); + } + async function withPackage(companyId: string, id: string, consume: (root: string, manifest: SkillPackageManifest) => Promise) { + const row = await descriptor(companyId, id); + const manifest = await manifestFor(row); + return withSkillVersionCacheLock(db, companyId, async () => { + const restored = await materialize(row, manifest); + touchSkillPackage(restored.directory); + if (restored.created) await trimSkillPackageCache(db, companyId, restored.directory, + (id, signal) => verifyDurable(companyId, id, signal).catch(() => false)); + return consume(restored.directory, manifest); + }); + } + async function files(companyId: string, id: string): Promise { + return withPackage(companyId, id, async (root, manifest) => { + const result: CompanySkillVersionFileInventoryEntry[] = []; + for (const entry of manifest.files) result.push({ path: entry.path, kind: entry.kind, content: (await fs.readFile(path.join(root, entry.path))).toString(entry.encoding), + ...(entry.encoding === "base64" ? { encoding: entry.encoding } : {}), ...(entry.executable ? { executable: true } : {}) }); + return result; + }); + } + async function prepare(companyId: string, files: CompanySkillVersionFileInventoryEntry[], migrateLegacy = false, cancellation?: AbortSignal, restore?: SkillPackageRestoreRecipe): Promise { + cancellation?.throwIfAborted(); + const manifest = skillPackageManifest(files); + if (restore && !skillPackageManifestBytes(restore.manifest).equals(skillPackageManifestBytes(manifest))) throw new Error("Restore recipe does not describe these package bytes"); + return flight(db, `prepare:${companyId}:${manifest.contentHash}`, async () => { + const signal = cancellation ? AbortSignal.any([cancellation, AbortSignal.timeout(180_000)]) : AbortSignal.timeout(180_000); + const token = randomUUID(); + const prefix = `${companyId}/skill-packages/${token}`; + const upload = { token, legacy: false, provider: storage().provider, archiveKey: `${prefix}/files.tar.gz`, manifestKey: `${prefix}/manifest.json`, expiresAt: new Date(Date.now() + 10 * 60_000).toISOString() }; + const [created] = await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); + return tx.insert(snapshots).values({ companyId, contentHash: manifest.contentHash, upload, restore, + migrationAt: null, maintenanceAt: new Date(Date.now() + 24 * 60 * 60_000) }).onConflictDoNothing().returning(descriptorColumns); + }); + let existing = created ?? (await db.select(descriptorColumns).from(snapshots).where(and(eq(snapshots.companyId, companyId), eq(snapshots.contentHash, manifest.contentHash))))[0]!; + if (!created) { + if (existing.storage || (!existing.upload && !migrateLegacy && !existing.restorable)) { + // Content deduplication must not attach a recipe with different kind or + // encoding metadata to an already published manifest. + if (restore && !existing.restorable && existing.storage?.manifestSha256 === packageSha256(skillPackageManifestBytes(manifest))) { + await db.update(snapshots).set({ restore }).where(and(scope(companyId, existing.id), sql`${snapshots.restore} is null`)); + } + return existing.id; + } + // A stale claim is replaced atomically; the DB trigger retains its exact + // keys in the deletion outbox. Never reuse keys from another attempt. + upload.legacy = !existing.upload || Boolean(existing.upload.legacy); + const [claimed] = await db.update(snapshots).set({ upload }).where(and(scope(companyId, existing.id), + sql`${snapshots.storage} is null`, + sql`(${snapshots.upload} is null or (${snapshots.upload}->>'expiresAt')::timestamptz <= now())`)).returning(descriptorColumns); + if (!claimed) throw conflict("This skill package is being saved. Retry after its upload settles."); + existing = claimed; + } + return withPackageIo(async () => { + const root = skillPackagePaths(companyId, existing.id).parent; + await ensureSkillCacheDirectory(root, true); + const staging = await fs.mkdtemp(path.join(root, ".upload-")); + const archivePath = path.join(staging, "files.tar.gz"); + const manifestBuffer = skillPackageManifestBytes(manifest); + if (manifestBuffer.length > MAX_SKILL_MANIFEST_BYTES) throw new Error("Skill manifest exceeds its byte bound"); + const backend = storage(); + try { + const archive = await writeSkillPackageArchive(archivePath, files, signal); + await backend.putFile({ companyId, namespace: "skill-packages", objectKey: upload.archiveKey, originalFilename: "files.tar.gz", + contentType: "application/gzip", body: createReadStream(archivePath), byteSize: archive.bytes, sha256: archive.sha256, signal }); + await backend.putFile({ companyId, namespace: "skill-packages", objectKey: upload.manifestKey, originalFilename: "manifest.json", + contentType: "application/json", body: manifestBuffer, signal }); + const receipt: SkillPackageStorageReceipt = { format: "tar-gzip-v1", provider: backend.provider, + archiveKey: upload.archiveKey, archiveSha256: archive.sha256, archiveBytes: archive.bytes, + manifestKey: upload.manifestKey, manifestSha256: packageSha256(manifestBuffer), manifestBytes: manifestBuffer.length, + fileCount: manifest.files.length, totalBytes: manifest.totalBytes }; + // A complete read verifies storage independently before publishing any reference. + await readBoundedObject(backend, companyId, receipt.manifestKey, receipt.manifestBytes, receipt.manifestSha256, signal); + const verify = await backend.getObject(companyId, receipt.archiveKey, { signal }); + const digest = boundedHashStream(receipt.archiveBytes); + const downloaded = path.join(staging, "verified.tar.gz"); + await pipeline(verify.stream, digest.stream, createWriteStream(downloaded, { flags: "wx", mode: 0o600 }), { signal }); + const actual = digest.result(); + if (actual.bytes !== receipt.archiveBytes || actual.sha256 !== receipt.archiveSha256) throw new Error("Uploaded skill package checksum mismatch"); + const restored = path.join(staging, "restored"); + await fs.mkdir(restored); + await extractSkillPackageArchive(downloaded, restored, manifest, signal); + signal.throwIfAborted(); + const [saved] = await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); + return tx.update(snapshots).set({ storage: receipt, upload: null, ...(restore ? { restore } : {}) }) + .where(and(scope(companyId, existing.id), sql`${snapshots.upload}->>'token' = ${token}`)).returning({ id: snapshots.id }); + }); + if (!saved) throw conflict("Skill package upload ownership changed"); + return saved.id; + } finally { + // Failed/ambiguous uploads retain their durable claim. Maintenance reconciles + // exact owned keys; never delete a successful object on an uncertain SQL result. + await fs.rm(staging, { recursive: true, force: true }); + } + }, signal); + }); + } + async function verifyDurable(companyId: string, id: string, cancellation?: AbortSignal): Promise { + const action = async () => { + const row = await descriptor(companyId, id); + if (!row.storage) { await manifestFor(row); if (!row.storage) return true; } + if (row.storage.provider !== storage().provider) throw new Error("Skill package storage provider changed; migrate its objects before switching storage"); + const signal = cancellation ? AbortSignal.any([cancellation, AbortSignal.timeout(180_000)]) : AbortSignal.timeout(180_000); + return withPackageIo(async () => { + const receipt = row.storage!; + const bytes = await readBoundedObject(storage(), companyId, receipt.manifestKey, receipt.manifestBytes, receipt.manifestSha256, signal); + const manifest = validateSkillPackageManifest(JSON.parse(bytes.toString("utf8"))); + if (manifest.contentHash !== row.contentHash) return false; + const parent = skillPackagePaths(companyId, id).parent; + await ensureSkillCacheDirectory(parent, true); + const staging = await fs.mkdtemp(path.join(parent, ".verify-")); + try { + const object = await storage().getObject(companyId, receipt.archiveKey, { signal }); + const digest = boundedHashStream(receipt.archiveBytes); + const archive = path.join(staging, "files.tar.gz"); + await pipeline(object.stream, digest.stream, createWriteStream(archive, { flags: "wx", mode: 0o600 }), { signal }); + const actual = digest.result(); + if (actual.bytes !== receipt.archiveBytes || actual.sha256 !== receipt.archiveSha256) return false; + const restored = path.join(staging, "files"); + await fs.mkdir(restored); + await extractSkillPackageArchive(archive, restored, manifest, signal); + return true; + } finally { await fs.rm(staging, { recursive: true, force: true }); } + }, signal); + }; + const context = readScope.getStore(); + const key = `${companyId}:${id}`; + if (context?.db !== db) return action(); + if (!context.durable.has(key)) context.durable.set(key, action()); + return context.durable.get(key)!; + } + async function versionFiles(version: StoredSkillVersion): Promise { + if (!version.packageSnapshotId) { + const [row] = await db.select({ ...skillVersionMetadataColumns, files: companySkillVersions.fileInventory }).from(companySkillVersions).where(and( + eq(companySkillVersions.companyId, version.companyId), eq(companySkillVersions.id, version.id))); + if (row?.packageSnapshotId) return versionFiles(row); + return row?.files ?? []; + } + if (!version.packageFileMap) return files(version.companyId, version.packageSnapshotId); + return withPackage(version.companyId, version.packageSnapshotId, async (root, manifest) => { + const byPath = new Map(manifest.files.map(file => [file.path, file])); + const result: CompanySkillVersionFileInventoryEntry[] = []; + for (const mapping of version.packageFileMap!) { + assertSkillSnapshotPath(mapping.path); + assertSkillSnapshotPath(mapping.sourcePath); + const file = byPath.get(mapping.sourcePath); + if (!file) throw new Error("Skill version references a missing package file"); + result.push({ path: mapping.path, kind: mapping.kind, + content: (await fs.readFile(path.join(root, file.path))).toString(file.encoding), + ...(file.encoding === "base64" ? { encoding: file.encoding } : {}), ...(file.executable ? { executable: true } : {}) }); + } + return result; + }); + } + return { descriptor, prepare, files, version, prefetch, versionFiles, withPackage, verifyDurable, + manifest: async (companyId: string, id: string) => manifestFor(await descriptor(companyId, id)) }; +} diff --git a/server/src/services/skill-package-work.test.ts b/server/src/services/skill-package-work.test.ts new file mode 100644 index 00000000000..ec8c8c1e699 --- /dev/null +++ b/server/src/services/skill-package-work.test.ts @@ -0,0 +1,102 @@ +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { eq, sql } from "drizzle-orm"; +import { companies, companySkillRepositorySnapshots as snapshots, createDb } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { createStorageService } from "../storage/service.js"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { skillPackageWork } from "./skill-package-work.js"; +import { skillPackageStore } from "./skill-package-store.js"; +import { skillSnapshotHash } from "./skill-snapshot.js"; +import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "./task-admission.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("automatic skill package work", () => { + let database: Awaited>; + let db: ReturnType; + let storage: ReturnType; + let coordinator: ReturnType | undefined; + let home: string; + const previousHome = process.env.PAPERCLIP_HOME; + const files = [{ path: "SKILL.md", kind: "skill" as const, content: "---\nname: legacy\n---\noriginal" }]; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("skill-package-work-"); + db = createDb(database.connectionString, { maxConnections: 2 }); + home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-package-work-")); + process.env.PAPERCLIP_HOME = home; + storage = createStorageService(createLocalDiskStorageProvider(path.join(home, "objects"))); + }); + beforeEach(async () => { + await db.execute(sql`truncate companies cascade`); + await db.execute(sql`truncate skill_package_deletions`); + }); + afterEach(async () => { await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks(); }); + afterAll(async () => { + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; + await db?.$client.end({ timeout: 1 }); await database?.cleanup(); + if (home) await fs.rm(home, { recursive: true, force: true }); + }); + async function company() { + const id = randomUUID(); + await db.insert(companies).values({ id, name: "Migration", issuePrefix: "MIG" }); + return id; + } + it("stops querying after empty startup and retains a first committed wake through drain", async () => { + const companyId = await company(); + let draining = false; + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, onError: error => { throw error; } }); + const work = skillPackageWork(db, storage).start(coordinator); + await work.ready; + expect(coordinator.nextWakeAt()).toBeNull(); + let queries = 0; + const debug = db.$client.options.debug; + db.$client.options.debug = () => { queries++; }; + try { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + await vi.advanceTimersByTimeAsync(120_000); + expect(queries).toBe(0); + draining = true; + let id: string; + await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); + [id] = (await tx.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning({ id: snapshots.id })).map(row => row.id); + }); + await vi.advanceTimersByTimeAsync(0); + await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0)); + const baseline = queries; + await vi.advanceTimersByTimeAsync(60_000); + expect(queries).toBe(baseline); + draining = false; + work.wake(); + await vi.advanceTimersByTimeAsync(0); + await vi.waitFor(async () => { + const [row] = await db.select().from(snapshots).where(eq(snapshots.id, id!)); + expect(row!.fileInventory).toEqual([]); + expect(row!.storage).not.toBeNull(); + }); + } finally { db.$client.options.debug = debug; } + }); + it("keeps legacy contents on failure, persists a retry and resumes after restart", async () => { + const companyId = await company(); + const [row] = await db.insert(snapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const upload = vi.spyOn(storage, "putFile").mockRejectedValue(new Error("offline")); + const service = skillPackageWork(db, storage); + await service.tick(new AbortController().signal); + const [failed] = await db.select().from(snapshots).where(eq(snapshots.id, row!.id)); + expect(failed!.fileInventory).toEqual(files); + expect(failed!.migrationAt!.getTime()).toBeGreaterThan(Date.now()); + expect(await service.deadline()).toBe(failed!.migrationAt!.getTime()); + upload.mockRestore(); + await db.update(snapshots).set({ migrationAt: new Date(0), upload: { ...failed!.upload!, expiresAt: new Date(0).toISOString() } }).where(eq(snapshots.id, row!.id)); + await skillPackageWork(db, storage).tick(new AbortController().signal); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.id, row!.id)); + expect(saved!.fileInventory).toEqual([]); + expect(saved!.migrationAt).toBeNull(); + expect(await skillPackageStore(db, storage).files(companyId, row!.id)).toEqual(files); + }); +}); diff --git a/server/src/services/skill-package-work.ts b/server/src/services/skill-package-work.ts new file mode 100644 index 00000000000..792d1d477e9 --- /dev/null +++ b/server/src/services/skill-package-work.ts @@ -0,0 +1,155 @@ +import { readPaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils"; +import { promises as fs } from "node:fs"; +import { and, eq, isNotNull, isNull, lte, sql } from "drizzle-orm"; +import { companySkillRepositorySnapshots as snapshots, companySkillVersions as versions, + skillPackageDeletions, agents, type Db } from "@paperclipai/db"; +import { getStorageService, type StorageService } from "../storage/index.js"; +import { logger } from "../middleware/logger.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { validateSkillPackageManifest, skillPackageManifestBytes, packageSha256 } from "./skill-package-files.js"; +import { skillPackageMaintenance } from "./skill-package-maintenance.js"; +import { skillPackagePaths, skillPackageStorageEnabled } from "./skill-package-store.js"; +import { ensureSkillCacheDirectory, protectedVersions, SkillCacheBusy, withSkillVersionCacheLock } from "./skill-version-cache.js"; + +export const SKILL_PACKAGE_RETENTION_MS = 30 * 24 * 60 * 60_000; +const RETRY_MS = 60 * 60_000; +const PAGE_SIZE = 4; + +/** A finite upgrade plus persisted cleanup deadlines. No work is added to turn + * preparation; completed history has no recurring SQL/network obligation. */ +export function skillPackageWork(db: Db, storage?: StorageService) { + const maintenance = skillPackageMaintenance(db, storage); + const backend = () => storage ?? getStorageService(); + async function retire(id: string, signal: AbortSignal) { + const [row] = await db.select({ id: snapshots.id, companyId: snapshots.companyId, contentHash: snapshots.contentHash, createdAt: snapshots.createdAt, + storage: snapshots.storage, upload: snapshots.upload, restore: snapshots.restore, migrationAt: snapshots.migrationAt }) + .from(snapshots).where(eq(snapshots.id, id)); + if (!row) return; + signal.throwIfAborted(); + // Orphan deletion is delayed so an upload can publish its first version. + const removeLocal = async () => { + const paths = skillPackagePaths(row.companyId, id); + try { + await ensureSkillCacheDirectory(paths.parent); + await fs.rm(paths.directory, { recursive: true, force: true }); + await fs.rm(paths.manifest, { force: true }); + } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } + }; + if (await withSkillVersionCacheLock(db, row.companyId, async () => { + if (!(await maintenance.collectOrphans({ id, apply: true, signal })).removed) return false; + await removeLocal(); + return true; + }, true)) return; + const orphanAt = row.createdAt.getTime() + 24 * 60 * 60_000; + let next: Date | null = orphanAt > Date.now() ? new Date(orphanAt) : null; + if (row.upload) next = new Date(Math.max(Date.now() + RETRY_MS, new Date(row.upload.expiresAt).getTime(), row.createdAt.getTime() + 24 * 60 * 60_000)); + else if (row.restore && row.storage) { + const expires = row.createdAt.getTime() + SKILL_PACKAGE_RETENTION_MS; + if (expires > Date.now()) next = new Date(expires); + else if (!row.migrationAt) { + const manifest = validateSkillPackageManifest(row.restore.manifest); + if (row.restore.version !== 1 || manifest.contentHash !== row.contentHash + || packageSha256(skillPackageManifestBytes(manifest)) !== row.storage.manifestSha256) throw new Error("Invalid skill restore recipe; keep durable bytes"); + await withSkillVersionCacheLock(db, row.companyId, async () => { + const referenced = await db.select({ id: versions.id }).from(versions).where(and(eq(versions.companyId, row.companyId), + sql`(${versions.repositorySnapshotId} = ${id} or ${versions.packageSnapshotId} = ${id})`)); + const [current] = await db.execute(sql`select 1 from company_skills s where s.company_id = ${row.companyId} + and s.current_version_id in (select id from company_skill_versions where repository_snapshot_id = ${id} or package_snapshot_id = ${id}) + union all select 1 from company_skill_test_runs t where t.company_id = ${row.companyId} + and t.status in ('queued', 'running') and t.skill_version_id in + (select id from company_skill_versions where repository_snapshot_id = ${id} or package_snapshot_id = ${id}) limit 1`); + if (current) return; + const pinned = new Set(); + for (const agent of await db.select({ config: agents.adapterConfig }).from(agents).where(eq(agents.companyId, row.companyId))) { + for (const entry of readPaperclipSkillSyncPreference(agent.config ?? {}).desiredSkillEntries) if (entry.versionId) pinned.add(entry.versionId); + } + const active = await protectedVersions(db, row.companyId); + if (!active || referenced.some(version => active.has(version.id) || pinned.has(version.id))) return; + signal.throwIfAborted(); + // Metadata/recipe remain. Triggered deletion receipts retain the old + // object keys through the rollback grace; future reads reacquire the SHA. + await db.update(snapshots).set({ storage: null }).where(and(eq(snapshots.id, id), isNull(snapshots.upload))); + await removeLocal(); + }, true); + } + } + await db.update(snapshots).set({ maintenanceAt: next }).where(and(eq(snapshots.id, id), lte(snapshots.maintenanceAt, new Date()))); + } + async function tick(signal: AbortSignal) { + if (!skillPackageStorageEnabled()) return; + for (const table of [snapshots, versions]) { + const rows = await db.select({ id: table.id }).from(table).where(lte(table.migrationAt, new Date())).orderBy(table.migrationAt).limit(PAGE_SIZE); + for (const row of rows) { + signal.throwIfAborted(); + try { + if (table === snapshots) await maintenance.migrateSnapshots({ id: row.id, apply: true, clearLegacy: true, signal }); + else { + // Bundled releases intentionally retain their small inline contract. + const result = await maintenance.migrateVersions({ id: row.id, apply: true, clearLegacy: true, signal }); + if (!result.inspected) await db.update(versions).set({ migrationAt: null }).where(eq(versions.id, row.id)); + } + } catch (error) { + signal.throwIfAborted(); + await db.update(table).set({ migrationAt: new Date(Date.now() + RETRY_MS) }).where(and(eq(table.id, row.id), isNotNull(table.migrationAt))); + logger.warn({ packageId: row.id }, "Skill package conversion deferred; original contents retained"); + } + } + } + const due = await db.select({ id: snapshots.id }).from(snapshots).where(lte(snapshots.maintenanceAt, new Date())).orderBy(snapshots.maintenanceAt).limit(PAGE_SIZE); + for (const { id } of due) { + try { await retire(id, signal); } + catch (error) { + signal.throwIfAborted(); + await db.update(snapshots).set({ maintenanceAt: new Date(Date.now() + (error instanceof SkillCacheBusy ? 60_000 : RETRY_MS)) }).where(and(eq(snapshots.id, id), lte(snapshots.maintenanceAt, new Date()))); + if (!(error instanceof SkillCacheBusy)) logger.warn({ packageId: id }, "Skill package cleanup deferred"); + } + } + const deletions = await db.select({ id: skillPackageDeletions.id }).from(skillPackageDeletions).where(and( + eq(skillPackageDeletions.provider, backend().provider), lte(skillPackageDeletions.deleteAfter, new Date()))).limit(PAGE_SIZE); + for (const { id } of deletions) { + try { await maintenance.deleteObjects({ id, apply: true, signal }); } + catch (error) { + signal.throwIfAborted(); + await db.update(skillPackageDeletions).set({ deleteAfter: new Date(Date.now() + RETRY_MS) }).where(eq(skillPackageDeletions.id, id)); + logger.warn({ deletionId: id }, "Skill package object deletion deferred"); + } + } + } + async function deadline() { + if (!skillPackageStorageEnabled()) return null; + const [row] = await db.execute<{ at: string | Date | null }>(sql`select min(at) as at from ( + select min(migration_at) as at from company_skill_repository_snapshots + union all select min(migration_at) from company_skill_versions + union all select min(maintenance_at) from company_skill_repository_snapshots + union all select min(delete_after) from skill_package_deletions where provider = ${backend().provider} + ) deadlines`); + return row?.at ? new Date(row.at).getTime() : null; + } + function start(coordinator: Pick, "register">) { + let requested = 0, completed = -1, next: number | null = null; + let recovered = false; + return coordinator.register(DELIVERY_QUEUES.skillPackages, { + retryMs: 5_000, pausePassiveMaintenance: true, + onSettled: () => { requested++; }, + async run(signal) { + const generation = requested; + if (!recovered && skillPackageStorageEnabled()) { + // A package previously protected by a run/pin may now be unused. + // Reconsider it once on restart, not on an idle polling interval. + await db.update(snapshots).set({ maintenanceAt: new Date() }).where(and(isNull(snapshots.maintenanceAt), + sql`${snapshots.storage} is not null and ${snapshots.restore} is not null`, + lte(snapshots.createdAt, new Date(Date.now() - SKILL_PACKAGE_RETENTION_MS)))); + recovered = true; + } + await tick(signal); + next = await deadline(); + if (next !== null) next = Math.max(Date.now() + 1_000, next); + completed = generation; + }, + hasPending: async () => false, + nextRunAt: () => requested !== completed ? Date.now() + 5_000 : next, + }); + } + return { tick, deadline, start }; +} diff --git a/server/src/services/skill-source-sync.ts b/server/src/services/skill-source-sync.ts index ae31b25145b..7bba472a7de 100644 --- a/server/src/services/skill-source-sync.ts +++ b/server/src/services/skill-source-sync.ts @@ -103,7 +103,7 @@ function setupActor(setup: Setup): AuthorizationActor { onBehalfOfUserId: setup.refreshUserId === 'local-board' ? null : setup.refreshUserId, source: 'agent_key' } : { type: 'board', userId: setup.refreshUserId, source: setup.refreshLocalTrusted ? 'local_implicit' : 'session', companyIds: [setup.companyId] }; } -async function setupContext(db: Db, setup: Setup): Promise { +export async function skillSourceSetupContext(db: Db, setup: Setup): Promise { if (!setup.refreshUserId && !setup.refreshLocalTrusted) throw forbidden('Choose a GitHub connection to enable automatic updates.'); if (!setup.refreshLocalTrusted) { const [member] = await db.select({ id: companyMemberships.id }).from(companyMemberships) @@ -152,7 +152,7 @@ async function setupContext(db: Db, setup: Setup): Promise { } export function skillSourceSyncService(db: Db, options: { context?: (setup: Setup) => Promise } = {}) { - const contextFor = options.context ?? (setup => setupContext(db, setup)); + const contextFor = options.context ?? (setup => skillSourceSetupContext(db, setup)); const sources = skillSourceService(db); let running: Promise<{ imported: number; checked: number }> | null = null; const shutdown = new AbortController(); @@ -211,7 +211,7 @@ export function skillSourceSyncService(db: Db, options: { context?: (setup: Setu try { const result = await sources.create(binding.companyId, { repositoryUrl: scan.repositoryUrl, trackingRef: binding.trackingRef, commitSha: scan.commitSha, packageMode: scan.packageMode, selectedPaths: scan.candidates.map(candidate => candidate.path), - connectionId: scan.connectionId ?? binding.connectionId }, { ...context, projectBindingId: binding.id, projectBindingLeaseToken: token }, scan); + connectionId: scan.connectionId ?? binding.connectionId }, { ...context, signal: shutdown.signal, projectBindingId: binding.id, projectBindingLeaseToken: token }, scan); sourceId = result.source.id; } catch (error) { if (!(error instanceof Error && 'status' in error && error.status === 409)) throw error; diff --git a/server/src/services/skill-sources.ts b/server/src/services/skill-sources.ts index 7129d97052f..220e3647fb0 100644 --- a/server/src/services/skill-sources.ts +++ b/server/src/services/skill-sources.ts @@ -1,3 +1,5 @@ +import { skillPackageManifest } from "./skill-package-files.js"; +import { skillPackageStorageEnabled, skillPackageStore } from "./skill-package-store.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from './delivery-work-notifications.js'; import { reserveAutomaticSkillWork, settleAutomaticSkillWork } from './skill-source-refresh-budget.js'; import { skillSourceRepairAction } from './skill-source-problems.js'; @@ -20,6 +22,7 @@ import { skillSnapshotHash } from './skill-snapshot.js'; type Tx = Parameters[0]>[0]; type SourceRow = typeof sources.$inferSelect; export type SkillSourceContext = { + signal?: AbortSignal; actor: { type: 'user' | 'agent' | 'system'; userId?: string | null; agentId?: string | null }; setup?: { userId: string | null; agentId?: string | null; localTrusted: boolean; grantId?: string | null }; projectBindingId?: string; @@ -33,6 +36,27 @@ const scope = (companyId: string, id: string) => and(eq(sources.companyId, compa export function skillSourceService(db: Db) { const skills = companySkillService(db); + const packages = skillPackageStore(db); + async function preparePackages(companyId: string, scan: ScannedSkillSource, selected: string[], sourceId: string, signal?: AbortSignal) { + if (!skillPackageStorageEnabled()) return new Map(); + const candidates = scan.skills.filter(candidate => selected.includes(candidate.path) && !candidate.error); + const prepared = new Map(); + if (!candidates.length) return prepared; + if (scan.packageMode === 'repository') { + if (scan.repositoryPackage?.error || !scan.repositoryFiles?.length) throw unprocessable(scan.repositoryPackage?.error ?? 'Repository package has no files.'); + const id = await packages.prepare(companyId, scan.repositoryFiles, true, signal, { + version: 1, sourceId, repositoryId: scan.repositoryId, repositoryUrl: scan.repositoryUrl, + commitSha: scan.commitSha, skillPath: null, manifest: skillPackageManifest(scan.repositoryFiles), + }); + for (const candidate of candidates) prepared.set(candidate.path, id); + } else { + for (const candidate of candidates) prepared.set(candidate.path, await packages.prepare(companyId, candidate.files, true, signal, { + version: 1, sourceId, repositoryId: scan.repositoryId, repositoryUrl: scan.repositoryUrl, + commitSha: scan.commitSha, skillPath: candidate.path, manifest: skillPackageManifest(candidate.files), + })); + } + return prepared; + } async function row(companyId: string, id: string, database: Db | Tx = db) { const [source] = await database.select().from(sources).where(scope(companyId, id)); if (!source || source.removedAt) throw notFound('Skill source not found.'); @@ -98,20 +122,23 @@ export function skillSourceService(db: Db) { const identity = createHash('sha256').update(`${scan.repositoryId}:${sourceId}${revision ? `:${revision}` : ''}:${skillPath}`).digest('hex').slice(0, 16); return `github/${scan.repositoryId}/${identity}/${normalizeAgentUrlKey(name) || 'skill'}`; } - async function publish(tx: Tx, source: SourceRow, scan: ScannedSkillSource, selectedPaths: string[], excludedFolders: string[], context: SkillSourceContext, selectionReviewed: boolean): Promise> { + async function publish(tx: Tx, source: SourceRow, scan: ScannedSkillSource, selectedPaths: string[], excludedFolders: string[], context: SkillSourceContext, selectionReviewed: boolean, prepared: Map): Promise> { await tx.execute(sql`SET CONSTRAINTS paperclip_skill_runtime_changed DEFERRED`); const mode = scan.packageMode ?? 'skills'; if (mode === 'repository' && scan.skills.length > 0 && (scan.repositoryPackage?.error || !scan.repositoryFiles?.length)) { throw unprocessable(scan.repositoryPackage?.error ?? 'Repository package has no files. Scan it again before importing.'); } - let repositorySnapshotId: string | undefined; - const repositoryHash = mode === 'repository' && scan.repositoryFiles?.length && !scan.repositoryPackage?.error ? skillSnapshotHash(scan.repositoryFiles) : null; - if (repositoryHash) { - await tx.insert(companySkillRepositorySnapshots).values({ companyId: source.companyId, contentHash: repositoryHash, fileInventory: scan.repositoryFiles! }) - .onConflictDoNothing(); - const [snapshot] = await tx.select({ id: companySkillRepositorySnapshots.id }).from(companySkillRepositorySnapshots).where(and( - eq(companySkillRepositorySnapshots.companyId, source.companyId), eq(companySkillRepositorySnapshots.contentHash, repositoryHash))); - repositorySnapshotId = snapshot!.id; + let repositorySnapshotId = mode === 'repository' ? prepared.values().next().value : undefined; + const hasSelected = scan.skills.some(candidate => selectedPaths.includes(candidate.path) && !candidate.error); + const repositoryHash = mode === 'repository' && hasSelected ? skillSnapshotHash(scan.repositoryFiles!) : null; + if (repositoryHash && !repositorySnapshotId) { + // Compatibility writer for a rolling deployment. Never save an otherwise + // unused repository when no valid skill is selected. + const [saved] = await tx.insert(companySkillRepositorySnapshots).values({ companyId: source.companyId, + contentHash: repositoryHash, fileInventory: scan.repositoryFiles! }).onConflictDoNothing().returning({ id: companySkillRepositorySnapshots.id }); + repositorySnapshotId = saved?.id ?? (await tx.select({ id: companySkillRepositorySnapshots.id }) + .from(companySkillRepositorySnapshots).where(and(eq(companySkillRepositorySnapshots.companyId, source.companyId), + eq(companySkillRepositorySnapshots.contentHash, repositoryHash))))[0]!.id; } const previous = await tx.select().from(entries).where(and(eq(entries.companyId, source.companyId), eq(entries.sourceId, source.id))); const imported: CompanySkill[] = [], updated: CompanySkill[] = []; @@ -157,6 +184,9 @@ export function skillSourceService(db: Db) { if (changed) { const version = await skills.createVersion(source.companyId, skillId!, { label: `GitHub ${scan.commitSha.slice(0, 8)}` }, context.actor, { database: tx, skipInventoryRefresh: true, skill: installed, fileInventory: candidate.files, + packageSnapshotId: prepared.get(candidate.path), + packageFileMap: repositorySnapshotId && prepared.has(candidate.path) ? candidate.files.map(file => ({ path: file.path, kind: file.kind, + sourcePath: file.path === 'SKILL.md' ? candidate.path : path.posix.join(path.posix.dirname(candidate.path), file.path) })) : null, ...(repositorySnapshotId ? { repositorySnapshotId, repositorySkillPath: candidate.path } : {}), }); installed.currentVersionId = version.id; @@ -203,8 +233,11 @@ export function skillSourceService(db: Db) { await context.authorize('skills.import', { sourceType: 'git', sourceLocator: scan.repositoryUrl, skillKey: newSkillKey(scan, candidate.path, candidate.name, id) }); } + const prepared = await preparePackages(companyId, scan, input.selectedPaths, id, context.signal); + context.signal?.throwIfAborted(); const published = await db.transaction(async tx => { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); const [source] = await tx.insert(sources).values({ id, companyId, repositoryId: scan.repositoryId, repositoryUrl: scan.repositoryUrl, fullName: scan.fullName, packageMode: scan.packageMode ?? 'skills', trackingRef: context.projectBindingId && input.trackingRef === 'HEAD' ? 'HEAD' : scan.trackingRef, connectionId: scan.connectionId ?? input.connectionId ?? null, lastAttemptAt: new Date(), accessScope: context.projectBindingId ? 'projects' : 'company', importNewSkills: Boolean(context.projectBindingId), @@ -215,7 +248,7 @@ export function skillSourceService(db: Db) { if (!binding?.active || (context.projectBindingLeaseToken && binding.leaseToken !== context.projectBindingLeaseToken)) throw conflict('Project repository changed during import.'); await tx.update(projectSkillSources).set({ sourceId: source.id, status: 'ready', lastError: null }).where(eq(projectSkillSources.id, binding.id)); } - const result = await publish(tx, source, scan, input.selectedPaths, input.excludedFolders ?? [], context, true); + const result = await publish(tx, source, scan, input.selectedPaths, input.excludedFolders ?? [], context, true, prepared); return { ...result, source: await detail(companyId, id, tx) }; }); context.publishCommittedActivity?.(); @@ -289,11 +322,14 @@ export function skillSourceService(db: Db) { } } await authorizeScan(source, scan, selectedPaths, context); + const prepared = await preparePackages(companyId, scan, selectedPaths, id, options.signal ?? context.signal); + (options.signal ?? context.signal)?.throwIfAborted(); const published = await db.transaction(async tx => { if (!options.automatic) await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); const [locked] = await tx.select().from(sources).where(scope(companyId, id)).for('update'); if (!locked || locked.leaseToken !== token || locked.revision !== source.revision || locked.leaseExpiresAt! < new Date()) throw conflict('Source refresh expired or changed. Try again.'); - const result = await publish(tx, locked, scan, selectedPaths, selection?.excludedFolders ?? source.excludedFolders, context, Boolean(selection)); + const result = await publish(tx, locked, scan, selectedPaths, selection?.excludedFolders ?? source.excludedFolders, context, Boolean(selection), prepared); if (selection || scan.connectionId !== undefined) await tx.update(sources).set({ ...(selection ? { enabled: true } : {}), connectionId: scan.connectionId ?? connectionId, ...(source.refreshGrantId === null && reader.grantId ? { refreshGrantId: reader.grantId } : {}), ...(selection?.connectionId !== undefined || (!source.refreshUserId && !source.refreshLocalTrusted && context.setup) ? { refreshUserId: context.setup?.userId, refreshAgentId: context.setup?.agentId ?? null, refreshLocalTrusted: context.setup?.localTrusted ?? false, refreshGrantId: reader.grantId ?? null } : {}) }).where(scope(companyId, id)); @@ -383,6 +419,7 @@ export function skillSourceService(db: Db) { } async function remove(tx: Tx, copiedCount: number, deletedCount: number) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); // Stop both pending attachments and an in-flight refresh before removing tracking. await tx.update(projectSkillSources).set({ active: false, leaseToken: null, leaseExpiresAt: null }) .where(and(eq(projectSkillSources.companyId, companyId), eq(projectSkillSources.sourceId, id))); diff --git a/server/src/services/skill-version-cache.ts b/server/src/services/skill-version-cache.ts index 57787129e80..f5dbbc37985 100644 --- a/server/src/services/skill-version-cache.ts +++ b/server/src/services/skill-version-cache.ts @@ -1,9 +1,10 @@ +import { skillPackageStore, skillVersionMetadataColumns } from "./skill-package-store.js"; import { AsyncLocalStorage } from "node:async_hooks"; import { randomUUID } from "node:crypto"; import { constants, promises as fs } from "node:fs"; import path from "node:path"; import { and, eq, sql } from "drizzle-orm"; -import { companySkillVersions, companySkillRepositorySnapshots, heartbeatRuns, +import { companySkillVersions, heartbeatRuns, nativeRunFinalizations, environmentLeases, withDedicatedDbConnection, type Db } from "@paperclipai/db"; import { resolvePaperclipInstanceRoot } from "../home-paths.js"; import { assertSkillSnapshotPath } from "./skill-snapshot.js"; @@ -16,11 +17,6 @@ export const SKILL_VERSION_CACHE_IDLE_MS = 60 * 60 * 1000; const locks = new AsyncLocalStorage>(); const uuid = /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i; export class SkillCacheBusy extends Error {} -// Repository snapshots are immutable and referenced with restrictive foreign -// keys. Cache only validated entrypoint names, never large file contents, so an -// hourly sweep does not transfer the same repository for each skill/version. -const verifiedSnapshots = new WeakMap>>(); - /** Hold through publication AND durable run pinning, not just file creation. */ export async function withSkillVersionCacheLock(db: Db, companyId: string, action: () => Promise, tryOnly = false): Promise { const held = locks.getStore() ?? []; @@ -95,18 +91,8 @@ export async function lastUsedAt(filename: string): Promise { } } -// A terminal run can still own provider execution or recovery. Only fully settled -// runs release their pins; resume preparation reacquires the same cache lock. -export async function protectedVersions(db: Db, companyId: string): Promise | null> { - const rows = await db.select({ - id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, nativeIssueId: heartbeatRuns.nativeIssueId, - pins: sql`${heartbeatRuns.contextSnapshot}->'skillVersions'`, - // Avoid loading large execution inputs for runs with the modern pin map. - legacyInput: sql`case when ${heartbeatRuns.runtimeMode} = 'native' - and not coalesce(${heartbeatRuns.contextSnapshot} ? 'skillVersions', false) - then ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput' end`, - }) - .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), sql`( +export function skillRunNeedsFilesSql() { + return sql`( ${heartbeatRuns.status} not in ('succeeded', 'failed', 'timed_out', 'cancelled', 'interrupted') or ${heartbeatRuns.finishedAt} is null or ${heartbeatRuns.controllerLeaseExpiresAt} > now() @@ -119,7 +105,21 @@ export async function protectedVersions(db: Db, companyId: string): Promise | null> { + const rows = await db.select({ + id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, nativeIssueId: heartbeatRuns.nativeIssueId, + pins: sql`${heartbeatRuns.contextSnapshot}->'skillVersions'`, + // Avoid loading large execution inputs for runs with the modern pin map. + legacyInput: sql`case when ${heartbeatRuns.runtimeMode} = 'native' + and not coalesce(${heartbeatRuns.contextSnapshot} ? 'skillVersions', false) + then ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput' end`, + }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), skillRunNeedsFilesSql())); const versions = new Set(); for (const { pins, legacyInput, id, agentId, nativeIssueId } of rows) { if (legacyInput != null) { @@ -163,27 +163,27 @@ function validateInventory(inventory: CompanySkillVersionFileInventoryEntry[]) { } export async function hasRestorableSnapshot(db: Db, companyId: string, versionId: string) { - const [version] = await db.select().from(companySkillVersions).where(and( + const [version] = await db.select(skillVersionMetadataColumns).from(companySkillVersions).where(and( eq(companySkillVersions.companyId, companyId), eq(companySkillVersions.id, versionId))); if (!version) return false; - if (version.repositorySnapshotId) { - if (!version.repositorySkillPath) return false; - let cache = verifiedSnapshots.get(db); - if (!cache) { cache = new Map(); verifiedSnapshots.set(db, cache); } - const key = `${companyId}:${version.repositorySnapshotId}`; - const verified = cache.get(key); - if (verified) return verified.has(version.repositorySkillPath); - const [snapshot] = await db.select({ inventory: companySkillRepositorySnapshots.fileInventory }) - .from(companySkillRepositorySnapshots).where(and(eq(companySkillRepositorySnapshots.companyId, companyId), - eq(companySkillRepositorySnapshots.id, version.repositorySnapshotId))); - if (!snapshot || !validateInventory(snapshot.inventory)) return false; - const entrypoints = new Set(snapshot.inventory.filter(file => file.encoding !== "base64" - && repositorySkillFrontmatter(file.content)).map(file => file.path)); - if (cache.size >= 128) cache.delete(cache.keys().next().value!); - cache.set(key, entrypoints); - return entrypoints.has(version.repositorySkillPath); + const packageId = version.repositorySnapshotId ?? version.packageSnapshotId; + const store = skillPackageStore(db); + if (packageId) { + if (version.repositorySnapshotId && !version.repositorySkillPath) return false; + const manifest = await store.manifest(companyId, packageId); + const entrypoint = version.repositorySkillPath ?? version.packageFileMap?.find(file => file.path === "SKILL.md")?.sourcePath ?? "SKILL.md"; + const entry = manifest.files.find(file => file.path === entrypoint); + if (!entry) return false; + // Git packages intentionally permit byte eviction. Their immutable recipe, + // not a network probe during cleanup, is the restoration contract. + if ((await store.descriptor(companyId, packageId)).restorable) return true; + if (version.repositorySnapshotId && (entry.encoding === "base64" || !await store.withPackage(companyId, packageId, + async root => repositorySkillFrontmatter(await fs.readFile(path.join(root, entrypoint), "utf8"))))) return false; + // Never evict the only known-good bytes based on a permanent process cache. + return store.verifyDurable(companyId, packageId); } - return validateInventory(version.fileInventory) && version.fileInventory.some(file => file.path === "SKILL.md"); + const files = await store.versionFiles(version); + return validateInventory(files) && files.some(file => file.path === "SKILL.md"); } // Counts allocated bytes without following links, matching what df can reclaim. diff --git a/server/src/services/skill-version-retention.ts b/server/src/services/skill-version-retention.ts index ab507fd8ee2..f59da11ceb2 100644 --- a/server/src/services/skill-version-retention.ts +++ b/server/src/services/skill-version-retention.ts @@ -1,3 +1,4 @@ +import { withSkillPackageReadScope } from "./skill-package-store.js"; import { randomUUID } from "node:crypto"; import { promises as fs } from "node:fs"; import path from "node:path"; @@ -417,7 +418,7 @@ export function skillVersionRetentionService(db: Db, defaults: SkillVersionPrune async function pruneCompany(companyId: string, options: SkillVersionPruneOptions & { cursor?: CompanyCursor | null } = {}) { if (scanning.has(companyId)) return { ...emptyResult(), checked: 1, skipped: { scan_busy: 1 } }; scanning.add(companyId); - try { return await pruneCompanyPage(companyId, options); } + try { return await withSkillPackageReadScope(db, () => pruneCompanyPage(companyId, options)); } finally { scanning.delete(companyId); } } diff --git a/server/src/storage/local-disk-provider.ts b/server/src/storage/local-disk-provider.ts index df3ed649fa6..9ee950117f3 100644 --- a/server/src/storage/local-disk-provider.ts +++ b/server/src/storage/local-disk-provider.ts @@ -48,7 +48,7 @@ export function createLocalDiskStorageProvider(baseDir: string): StorageProvider const tempPath = `${targetPath}.tmp-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; try { - await fs.writeFile(tempPath, input.body); + await fs.writeFile(tempPath, input.body, { signal: input.signal }); await fs.rename(tempPath, targetPath); } finally { await fs.rm(tempPath, { force: true }); } }, @@ -66,7 +66,7 @@ export function createLocalDiskStorageProvider(baseDir: string): StorageProvider ? input.range.end - input.range.start + 1 : stat.size; return { - stream: createReadStream(filePath, streamOptions), + stream: createReadStream(filePath, { ...streamOptions, signal: input.signal }), contentLength, lastModified: stat.mtime, }; diff --git a/server/src/storage/s3-multipart.ts b/server/src/storage/s3-multipart.ts index 2b8e0d951d4..169b35d71e9 100644 --- a/server/src/storage/s3-multipart.ts +++ b/server/src/storage/s3-multipart.ts @@ -13,7 +13,7 @@ export async function putS3Multipart(client: S3Client, bucket: string, key: stri // Scale part sizes to the S3 part-count limit while retaining stream backpressure. const partSize = Math.max(8 * 1024 * 1024, Math.ceil(input.contentLength / 10_000)); try { - uploadId = (await withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType })))).UploadId; + uploadId = (await withS3FailureDiagnostics("create_multipart_upload", () => client.send(new CreateMultipartUploadCommand({ Bucket: bucket, Key: key, ContentType: input.contentType }), { abortSignal: input.signal }))).UploadId; if (!uploadId) throw new Error("S3 did not return a multipart upload id"); const parts: { ETag: string; PartNumber: number }[] = []; for (let offset = 0; offset < input.contentLength;) { @@ -35,14 +35,14 @@ export async function putS3Multipart(client: S3Client, bucket: string, key: stri } })()); try { - const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: parts.length + 1, Body: body, ContentLength: length }))); + const result = await withS3FailureDiagnostics("upload_part", () => client.send(new UploadPartCommand({ Bucket: bucket, Key: key, UploadId: uploadId, PartNumber: parts.length + 1, Body: body, ContentLength: length }), { abortSignal: input.signal })); if (remaining || !result.ETag) throw new Error("S3 upload part is incomplete"); parts.push({ ETag: result.ETag, PartNumber: parts.length + 1 }); } finally { body.destroy(); } offset += length; } if (pending.length || !(await iterator.next()).done) throw new Error("S3 upload source exceeds its declared size"); - await withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }))); + await withS3FailureDiagnostics("complete_multipart_upload", () => client.send(new CompleteMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId, MultipartUpload: { Parts: parts } }), { abortSignal: input.signal })); } catch (error) { if (uploadId) await client.send(new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })).catch(() => {}); throw error; diff --git a/server/src/storage/s3-provider.ts b/server/src/storage/s3-provider.ts index b96ff003be8..dce043fb8fb 100644 --- a/server/src/storage/s3-provider.ts +++ b/server/src/storage/s3-provider.ts @@ -95,6 +95,7 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid ContentType: input.contentType, ContentLength: input.contentLength, }), + { abortSignal: input.signal }, )); }, diff --git a/server/src/storage/service.ts b/server/src/storage/service.ts index b0020308556..ff821616e08 100644 --- a/server/src/storage/service.ts +++ b/server/src/storage/service.ts @@ -107,6 +107,7 @@ export function createStorageService(provider: StorageProvider): StorageService body: input.body, contentType, contentLength: byteSize, + signal: input.signal, }); } catch (error) { await provider.deleteObject({ objectKey }).catch(() => {}); @@ -127,7 +128,7 @@ export function createStorageService(provider: StorageProvider): StorageService async getObject(companyId: string, objectKey: string, options) { ensureCompanyPrefix(companyId, objectKey); - return provider.getObject({ objectKey, range: options?.range }); + return provider.getObject({ objectKey, range: options?.range, signal: options?.signal }); }, async headObject(companyId: string, objectKey: string) { diff --git a/server/src/storage/types.ts b/server/src/storage/types.ts index 36d941147f8..9e7d1515ebc 100644 --- a/server/src/storage/types.ts +++ b/server/src/storage/types.ts @@ -3,6 +3,7 @@ import type { Readable } from "node:stream"; export interface PutObjectInput { objectKey: string; + signal?: AbortSignal; // Readable bodies stream straight to the backend (contentLength must be the // exact byte size); Buffer stays supported for small payloads. body: Buffer | Readable; @@ -45,6 +46,7 @@ export interface StorageProvider { } export type PutFileInput = { + signal?: AbortSignal; /** Server-allocated, company-prefixed key for durable idempotent uploads. Never accept from client input. */ objectKey?: string; companyId: string; @@ -65,7 +67,7 @@ export interface PutFileResult { export interface StorageService { provider: StorageProviderId; putFile(input: PutFileInput): Promise; - getObject(companyId: string, objectKey: string, options?: Pick): Promise; + getObject(companyId: string, objectKey: string, options?: Pick): Promise; headObject(companyId: string, objectKey: string): Promise; deleteObject(companyId: string, objectKey: string): Promise; }