## Thinking Path > - Paperclip manages agents and the skills they need to work. > - First-agent setup resolves the bundled skills before onboarding can finish. > - Skill packages use durable upload claims to protect their object-storage receipts. > - A failed attempt kept its claim active for ten minutes. A concurrent caller received an immediate conflict. > - This pull request expires a failed owner's unpublished claim and waits for an active upload to settle. > - Onboarding can retry skill setup without a false upload-in-progress error. ## Linked Issues or Issue Description **What happened?** First-agent onboarding could show "This skill package is being saved. Retry after its upload settles." during the subscription connection step. The agent's bundled skill setup called the package store. A failed upload retained its active claim, and calls from another database client also returned a conflict while an upload was in progress. **Expected behavior** A failed attempt permits an immediate retry with new object keys. Concurrent callers reuse the completed receipt or claim the failed attempt. They preserve another owner's active lease and every published receipt. **Steps to reproduce** 1. Fail the first object upload while resolving the first agent's bundled skills. 2. Retry the same skill selection before the ten-minute lease expires. 3. Observe the upload-in-progress conflict. The new regression reproduces this through `companySkillService.resolveRequestedSkillEntries`. **Paperclip version or commit** Reproduced on `6bf312270c3e`. **Deployment mode** Hosted onboarding; the package-store fix also applies to self-hosted instances. Related implementation: #18082. Searched open and closed pull requests and open issues for related skill-upload and lease failures. No duplicate fix was found. ## What Changed - Expire only the failed attempt's token-matched, unpublished upload claim. Keep its exact keys for the existing deletion outbox. Cleanup has a one-second client deadline and a transaction-local statement timeout. Queued cleanup skips its write after that deadline; failed cleanup retains the original crash lease. - Wait for a competing package preparation within the existing three-minute deadline. Re-read the receipt every 250 ms while this concrete request remains active. Release the database connection between reads and stop on cancellation. This adds no idle worker or permanent polling. - Preserve cancellation reasons, published receipts after lost commit replies, and claims acquired by a newer owner. - Add real PostgreSQL regression coverage for first-agent bundled skill setup, cross-client contention, failed-owner takeover, crash expiry, cancellation, ambiguous publication, and ownership changes. - Document upload recovery behavior. No schema or public API change is required. ## Verification - Red: four new or updated checks failed against the original implementation. They reproduced the exact conflict and the retained ten-minute lease. - Green: 189 targeted tests passed across `skill-package-upload.test.ts`, `skill-package-store.test.ts`, `skill-package-work.test.ts`, both package-storage migration suites, and `agent-skills-routes.test.ts`. The combined run passed 188 tests. The final SQL-lock regression passed separately. - Command: `pnpm exec vitest run server/src/__tests__/skill-package-upload.test.ts server/src/__tests__/skill-package-store.test.ts server/src/services/skill-package-work.test.ts server/src/services/skill-package-storage-migration.test.ts server/src/services/skill-package-storage-migration-postgres.test.ts server/src/__tests__/agent-skills-routes.test.ts`. - `pnpm --filter @paperclipai/server typecheck` passed, including source and test layers. - The cancellation test first exposed Node's timer wrapping the caller's abort reason. The final implementation preserves that reason, and the full targeted run passed. - Greptile identified an unbounded cleanup write. Two new real-PostgreSQL tests reproduced delayed errors and cancellations behind an occupied one-connection pool. They now pass and prove that late cleanup skips its write. A further row-lock test proves that the PostgreSQL statement timeout frees the pool while another transaction still holds the package row. - Local setup initially lacked macOS embedded-Postgres library links. Repaired ignored dependency links before the red and green database tests. The initial server typecheck also lacked SDK links for excluded plugin packages; ran the maintained `node scripts/link-plugin-dev-sdk.mjs` helper before repeating it. - `pnpm verify --explain` returned exit 2 and requested broader CI coverage. No full local suite was substituted for that CI requirement. - `git diff --check` passed. Reviewed the full diff for secrets, private URLs, personal data, generated files, and run artifacts. - Final revision: `4b4814ac01522599901a3dc29f650a1b33c0b4e2`. [CI run](https://github.com/paperclipai/paperclip/actions/runs/44880245673) passed all applicable checks, including typecheck, build, all general and serialized server test shards, all e2e shards, Runner verification, and the final verify gate. [Greptile review](https://github.com/paperclipai/paperclip/pull/22477#issuecomment-7059199437) scored this revision 5/5 with no unresolved threads. GitHub reports the branch as mergeable and clean. ## Risks Contended preparations now wait within their existing request deadline. A crashed process or an unavailable cleanup database still uses the ten-minute lease expiry. Genuine storage failures remain errors; this change does not hide an unavailable backend. The existing object deletion grace remains intact. Live provider authorization and a deployed Cloud onboarding journey were not run. Tenant release and deployment are separate from this source fix. No production migration is needed. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh reasoning effort and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
23 KiB
Installed skill package storage
Installed repository and editable-skill contents can live in the configured storage backend instead of PostgreSQL JSONB. The database keeps company-scoped package identities, storage receipts, source tracking, versions, permissions and run pins. The original repository tree, wrappers, binary bytes and executable flags retain their existing behavior. New Git packages also retain an immutable restore recipe; locally authored/edited packages and unproven legacy packages keep a durable copy.
Installed content authority
Paperclip-owned skills use sourceType: managed, sourceLocator: null, and an
immutable currentVersionId. Creation, edits, deletion of files, forks and restores
prepare verified packages before a short transaction publishes the version, head,
metadata and audit. A competing head returns 409; it never overwrites another
writer's files. Acknowledged saves survive cache loss. Studio submits the version it
read and preserves unsaved text on a conflict.
Repository and catalog installations also read committed versions. Runtime selection
persists version pins before delivery materializes files. Recovery uses the pinned
version and checks its reconstructed v1 asset digest; a previous server's rootPath
is not proof of availability. Active or recovering run pins block whole-skill deletion,
even after the agent assignment is detached. Explicit linked-local sources retain
their local filesystem behavior and are outside this replacement guarantee.
See ephemeral web-server operation for the one-time installation conversion, rollback boundary and qualification procedure.
Automatic package upgrade
No data-migration command is required. The normal schema upgrade applies migration
0415 (0415_zippy_zemo.sql). External writes and automatic conversion are enabled by default. Startup
restores pending work from indexed migration_at / maintenance_at columns. The
shared delivery coordinator converts four snapshots and four versions per batch,
reading one legacy body at a time. It verifies the hash, uploads, reads the objects
back, validates extraction, then clears the legacy JSONB. Version IDs, revision numbers
and run pins stay unchanged. Failures retain the original bytes and retry after an hour.
Bundled release inventories keep their existing small inline validation contract.
Readers can use legacy contents while conversion is pending, including when conversion commits between the metadata read and the content read. Conversion runs separately from turn preparation. Completed migration history has no recurring scan. Normal vacuum and backup retention determine when PostgreSQL reclaims physical space.
PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED=false is an explicit compatibility override
for a coordinated rolling upgrade: it disables new external writes and automatic
package maintenance. All processes must support the new reader before allowing
external-only writes or clearing old payloads. Once automatic conversion runs, this
reader is the rollback floor; an older binary needs rehydrated database contents.
Use durable, backed-up local storage or S3-compatible storage for authored packages.
The extracted cache is not that storage backend.
Managed file and folder deletion publishes a new inventory. It requires no local rollback copy. Unused uploads follow the existing orphan/deletion machinery; a post-commit cache or notification failure cannot undo a committed save. Only the explicit linked-local compatibility adapter retains filesystem rollback handling.
Create idempotency receipts retain the original response projection and immutable
version identity in company_skill_mutation_receipts; file-save receipts keep their
existing activity receipt shape. Retries reauthorize access. Deleted identities do
not transfer their receipts to a skill recreated with the same name. Skill bodies
remain in packages/versions rather than activity logs.
The optional server/scripts/migrate-skill-packages.ts command remains available for
diagnostics and controlled batches. It defaults to dry run, supports
--phase=snapshots|versions|orphans|deletions|installations|eligibility|verification, --company=<uuid>, --limit=1..50,
--after=<uuid>, --apply and --clear-legacy. It is not part of normal installation
or compatible routine upgrade. The installations phase is the one-time legacy
conversion boundary described in the linked runbook. Never point it at another instance's database/storage configuration.
New package-backed versions retain their complete path/kind/encoding index in
package_file_map, including identity mappings. Library and version listing read
this metadata without downloading manifests or archives. The bounded versions
phase also backfills older null indexes from verified manifests without changing
version IDs or content; include that phase in legacy conversion. Old null indexes
remain readable during conversion, with a cold manifest read when needed.
Acquisition and retention
A missing package follows this order: verified local files, stored archive, legacy DB contents, then an authorized exact-commit Git fetch. A new Git recipe records repository identity, the original source, a full commit SHA, packager version, the precise included file manifest (paths, encodings, modes, sizes and hashes), and the content hash. It is loaded only on a cold miss, never with warm metadata queries. Recovery reads exactly those files without rerunning discovery or today's filtering rules. A reused repository name, wrong commit, changed file/mode/hash, missing original commit, or revoked source credentials fails explicitly; it never substitutes the latest branch. Concurrent misses coalesce and failed acquisition has a persisted five-minute backoff.
After 30 days, unused Git packages with a verified recipe can lose their cached archive and extracted copy while retaining their identity, manifest and recipe. Current library versions, explicit agent pins, active runs and unresolved recovery keep their packages. A source/version change schedules another eligibility check; packages held by a run or pin are also reconsidered on restart. This is conservative retention, not an immediate promise to purge on unassignment. Legacy packages without a provable recipe and authored or edited copies retain their durable bytes as long as a version references them. A shared package remains durable when any referencing version lacks its own immutable repository reference. This also conservatively retains per-skill Git imports without repository projections. Authored preparation permanently marks the storage receipt as durable, including when it joins a concurrent Git upload. Expiry checks that marker in its update, so a later Git import cannot weaken an authored version's retention.
Native runtime assets expire after 30 days without use. An hourly coalesced filesystem access lease protects preparation through durable run pinning. Cleanup shares the publication lock, preserves assets referenced by active/recovering runs, and defers if an unfinished input cannot be interpreted safely. After a pin blocks cleanup, the next asset use or startup reconsiders it. Missing or damaged copies rebuild from verified source bytes. The canonical package cache retains its capacity policy below, and runtime version directories retain their existing pin-aware retention.
Package/version publication and deletion use transaction-aware work notifications; local asset use signals its filesystem worker. Both reuse the delivery coordinator, restore deadlines on startup, pause in drain/standby, and stop on shutdown. With no pending conversion, retry, or eligible cache deadline they perform no recurring work. Compatible servers resolve the durable head on each content operation and use transactional publication checks. Content correctness needs no process-local invalidation. Maintenance wake delivery across instances remains owned by the delivery coordinator; startup reconstructs pending deadlines. No permanent skills-specific DB listener is added. Conversion retains authoritative legacy bytes until verification succeeds. Conversion retries and cache/deletion dates are optional maintenance eligibility; they may wait until the next ingress wake, including with package storage disabled. They do not block idle sleep or claim an external Cloud alarm or measured host suspension. Each object-deletion receipt has a 30-second storage deadline. Shutdown cancels pending delete/existence requests; incomplete cleanup retains the receipt for retry.
Execution and cache behavior
- New packages store one gzip tar archive and one manifest per company/content hash. The manifest contains paths, kinds, encodings, executable flags, byte sizes and SHA-256 values. PostgreSQL stores keys, hashes, sizes and ownership metadata.
- Preparation uses metadata-only version/package queries. Multiple selected versions are batched, and a shared repository descriptor is read once per preparation.
- Known selected repository skills skip unrelated library reconciliation. Selected
rows and project visibility are still read fresh. Public rows are constrained with
access_source_id IS NULLat the final read, so a concurrent scope change fails closed. Aliases, legacy operational skills, missing installations and mutable local sources retain the discovery/reconciliation path. - A cold package download is coalesced in process; the existing company cache lock serializes cross-process restoration and pinning. The archive and each extracted file are verified before atomic publication. Extraction rejects unexpected entries, path escapes, links, duplicate paths, wrong sizes, modes and digests.
- Warm checks inspect directory entries and file identity/size/mode/mtime/ctime. Unchanged files avoid content reads and hashing. Changed identities are hashed again; altered runtime copies are rebuilt. No marker alone establishes integrity.
- Native asset packaging also caches manifests and asset references, checking both the selected source and delivered asset on reuse. It retains no repository bodies. Source edits invalidate the reference; missing or corrupt copies rebuild from verified bytes.
- Concurrent cold package I/O is limited to two operations, with a queue of at most 32. Cached version inventories and file previews have a separate two-reader, 32-waiter bound, so slow storage cannot prevent verified local reads. Both queues honor deadlines and cancellation; an overloaded local-read queue does not trigger a storage download. Identical cold version/file reads are coalesced. Verified scratch restores repopulate the shared cache under a short, nonblocking publication lock, so subsequent reads reuse local bytes. Eviction verification runs outside that publication lock. Cold administrative responses also have a two-response, 32-waiter bound held through post-fill cleanup, so decoded inventories cannot accumulate behind storage work. Caller-scoped metadata is read before taking that permit. Acquisition and cleanup use a private database connection, so queued transaction readers cannot deadlock the ordinary pool, including instances configured with one connection. Legacy repository bodies load only inside the I/O limit, including when deriving a missing manifest. Waiting readers retain metadata rather than repository contents. Packages share the Git scanner's limits of 10,000 files and 100 MiB decoded bytes. Recovery splits reads within the Git reader's 9,990-file and 61 MiB batch limits. Individual manifests are limited to 48 MiB; the manifest cache holds 16 MiB of serialized data. Repository projection metadata is shared; generated wrappers/policies have a 16 MiB budget and 256-entry limit. The stat/digest cache holds at most 128,000 file records, with 24 concurrent file checks. Native skill asset manifests hold at most 32,000 file records and 128 entries.
- Shared extracted packages target 512 MiB and 64 directories per company, trimming on cold fills only, with at most two eviction attempts and a ten-second storage verification budget per fill. This is a soft limit during a storage outage: cleanup preserves the last recoverable copy. Version projections retain the existing pin-aware cleanup. Abandoned package staging directories receive a one-hour crash grace.
- Internal package caches, their files, and containing directories cannot be imported as local skill sources. Fork the installed skill to retain its original access restrictions.
Each entrypoint still has a self-contained runtime tree, and remote staging can duplicate those trees. This change does not introduce a new runner transport format or cross-entrypoint asset deduplication.
Publication, cleanup, and backups
Source imports prepare and verify packages before opening their publication transaction. Generic authoring calls can arrive inside an existing edit/idempotency transaction; these retain that serialization lock during upload and use an independent package connection to avoid a one-connection pool deadlock. No upload happens on the unchanged warm preparation path.
Uploads use unique attempt keys. An incomplete attempt has a durable claim with a
ten-minute crash expiry. When an attempt fails or is cancelled, it expires only its
own unpublished claim so a retry can use fresh keys immediately. This cleanup has
a one-second client deadline and a transaction-local statement timeout. Cleanup
queued behind a busy pool skips its write after that deadline. If the database is
unavailable, the original crash expiry remains. Concurrent preparation on another
connection or replica waits for the receipt or an expired claim within its existing
three-minute request deadline. These reads stop when the request settles or is cancelled;
they add no idle background polling. An ambiguous database result never triggers
deletion of a possibly successful upload. A database trigger records
retired keys in skill_package_deletions, including company cascades. Promotion of
an upload to a published receipt does not enqueue its objects for deletion.
The maintenance phases orphans and deletions have the same dry-run/cursor controls.
orphans only removes unreferenced snapshot rows older than one day whose upload
claim is settled. Foreign keys protect referenced versions. deletions rechecks that
no live receipt owns the keys, deletes the two objects, confirms absence, then removes
the deletion receipt. Failed attempts have at least one hour of grace; retired
published objects have 30 days. These phases also run automatically from their saved
deadlines. Do not apply a blanket bucket lifecycle rule: authored packages and legacy
packages without exact restore recipes still require durable retention.
Database backups must be paired with object backups/history. Do not process deletion receipts while a retained database backup outside the 30-day window still needs those objects; extend receipt deadlines or retain object versions according to that backup policy. A storage provider/bucket/prefix change requires copying and verifying the objects first. The current storage facade does not find objects in an old bucket.
Verification
skill-package-store.test.ts exercises real PostgreSQL/local storage, cold/warm
reads, same-size edits, offline historical restoration, missing durable objects,
upload recovery, company-deletion receipts, cancellation and resumable migration.
A reader that obtained its descriptor before the legacy clear pass follows the new
receipt, including when its old manifest is already cached. Existing source,
version-cache, retention, concurrency, authoring and runtime-context suites preserve
their original behavior checks. runtime-context.test.ts checks warm asset reuse,
source edits and repair through the normal native context builder. Additional recovery,
worker-lifecycle and asset-retention tests cover exact-commit restoration, automatic
conversion, persisted retry, idle/drain behavior and rebuilding expired native copies.
The native preparation benchmark accepts --package-store=external and
--native-assets=true. It reports elapsed time, PostgreSQL wire response bytes,
root-pool query count, object GET count/bytes, process RSS and preparation spans.
Use --snapshot-file=<inventory.json> for a representative saved package,
--actor=agent --staging-shape=true for authorization and unrelated-library load,
and --network-delay-ms=5 for an emulated database delay. Synthetic local timings
are regression evidence, not a claim about deployed Railway latency.
To generate the same-size synthetic fixture without copying a private repository:
python3 - <<'PYTHON'
import hashlib, json
files = [{"path": "skills/diagnostic/SKILL.md", "kind": "skill", "content":
"---\nname: diagnostic\ndescription: Native preparation benchmark\n---\nUse the bundled repository.\n"}]
for i in range(1348):
text = hashlib.shake_256(str(i).encode()).hexdigest(8300)
content = "\n".join(text[n:n + 64] for n in range(0, len(text), 64))[:16600]
files.append({"path": f"assets/{i}.txt", "kind": "reference", "content": content})
with open("/tmp/skill-package-benchmark.json", "w") as output:
json.dump(files, output, separators=(",", ":"))
PYTHON
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-skill-preparation.ts --snapshot-file=/tmp/skill-package-benchmark.json --package-store=external --native-assets=true --actor=agent --staging-shape=true --network-delay-ms=5 --selected-skills=10 --warm-runs=20
The fixture generator reproduces the recorded file counts and JSON size, with new
synthetic content. Use --selected-skills=1, 2, or 10 for comparable cases.
Measured results, 2026-10-28
Recorded samples use a synthetic 1,349-file, 22,799,129-byte JSON inventory. The multi-skill cases add one entrypoint per additional selected skill. Each case includes native asset preparation, fresh agent authorization, 155 unrelated library skills, and 5 ms of database network delay in each direction. The table summarizes all 20 numbered warm samples; a fresh service instance in the same process was measured separately.
| Selected skills | Warm median | Warm p95 | PostgreSQL response bytes | Ordinary queries | Object GETs |
|---|---|---|---|---|---|
| 1 | 245 ms | 326 ms | 19,288 | 5 | 0 |
| 2 | 298 ms | 332 ms | 20,312 | 5 | 0 |
| 10 | 573 ms | 674 ms | 28,536 | 5 | 0 |
All warm cases transferred zero package file-content bytes through SQL and zero object bytes. Query counts exclude the dedicated advisory-lock connection; wire bytes include the measured proxy traffic. Cold preparation fetched one manifest and one archive, about 13.45 MB total, even with ten selected entrypoints. Cold elapsed times were 3.7, 6.3 and 11.0 seconds respectively. These local samples ran alongside other verification work and do not establish production latency or a CPU reduction against an unchanged baseline.
Median process CPU per warm preparation was 83/175/820 ms for 1/2/10 skills; filesystem checks run concurrently, so CPU can exceed wall time. Process peak RSS was about 887/937/1,235 MB, including the application, synthetic fixture and cold allocations; these are not incremental cache-memory measurements. Allocated instance file bytes were about 97/152/596 MB. Runtime/native tree duplication remains visible here and is a separate optimization. Cache budgets also mean sufficiently large selected sets or many active companies can evict metadata/digests and require new hashing or asset assembly; the warm timing is not a constant-time guarantee for arbitrary workloads.
The real S3-compatible proof used a disposable loopback MinIO server built from its
official RELEASE.2025-04-22T22-12-26Z source. An 18,008,626-byte archive containing
22 files (18,000,095 decoded bytes) passed multipart upload, independent read-back
verification and cold restoration. server/scripts/verify-skill-package-s3.ts repeats
this test against a loopback endpoint and creates/deletes its own bucket and database.
No production storage or Railway instance was changed.
Workspace typecheck, build, migration checks and focused integration tests passed. The attempted full repository test run did not complete cleanly: parallel shards reported fixture setup timeouts and assertions, then were stopped for isolation. The affected skill concurrency assertion and sampled recovery/workspace failures passed on isolated reruns. The remaining broad inventory is not claimed as verified; see the implementation record.
Automatic lifecycle follow-up verification
The final implementation passed 30 focused package/recovery/worker/native tests, then 15 package/worker tests including two additional regressions for missing stored objects with a live receipt and retaining unproven historical bytes. Source-sync, version-retention and native integrations passed 73 tests. The idle-safety suite also passed. Full workspace typecheck/build passed; the final server was rebuilt after the manifest normalization fix. Migrations and map checks passed.
The shared PostgreSQL test-template path timed out (and exited a standalone benchmark
before measurement). The passing final runs used the supported
PAPERCLIP_TEST_POSTGRES_TEMPLATE=0 fresh-cluster path. Earlier retirement assertions
caught PostgreSQL JSONB key reordering; manifest serialization now preserves its v1
wire encoding after a DB round trip. An empty native cache startup case was also fixed.
The final 20-sample warm runs measured:
| Selected skills | Median | p95 | SQL response bytes | Ordinary queries | Object reads |
|---|---|---|---|---|---|
| 1 | 256 ms | 302 ms | 19,423 | 5 | 0 |
| 10 | 691 ms | 816 ms | 28,969 | 5 | 0 |
All warm samples still transfer zero package bodies through SQL and zero object bytes. The larger restore recipe stays out of those queries. Ten-skill cold preparation was slower on this busy host; the raw receipts retain that result as well as warm samples. The earlier full repository test attempt remains incomplete; this follow-up does not claim a clean full-suite result, deployed Railway measurements, or a live agent run.
Administrative version reads
Version lists, current-version summaries, and test-run detail return file names/kinds without bodies. Diff and restore fetch only the selected version through its existing single-version endpoint. Archive reads for those operations and retention checks use isolated scratch directories, so storage latency does not hold the company preparation lock. Support inspection reads the requested stored version without Git recovery.
Package preparation renews a 24-hour publication grace. Publication and retirement lock the same package row and recheck its receipt/current references before commit; a prepared receipt retired before publication cannot silently become current.