PaperClipAI/doc/skill-ephemeral-web-servers.md

# Skills on ephemeral web servers

The skills guarantee is replacement of A by B, and then C, with empty web-server
homes and the same database, object storage, secrets and supported application
artifacts. Disk remains useful for scratch archives, verified caches and transport
staging. It does not own acknowledged managed edits or installed version selection.
This guarantee does not qualify instructions, workspaces, controller state, run
logs, traffic switching or readiness/draining for volume removal.

## Ownership and publication

`managed` is Paperclip ownership. Its locator is null; editability does not depend
on a directory's position beneath a managed root. Existing policies normalize it
to `workspace`, the policy category previously used by managed local skills.
Repository provenance and tracking remain separate from installed content. Forks
and disconnected editable copies retain source access and cannot publish restricted
content through a public source category.

`managed-skill-content.ts` reads immutable version inventories, prepares candidates,
and publishes them against a captured head. Package upload and verification happen
before the final database transaction. That transaction rechecks identity/access,
locks names in sorted order, locks the skill, verifies the base, and publishes the
version, head, projections and applicable mutation audit/receipt. Publication races
return `409`; clients must reconcile instead of automatically rebasing. Complete
inline versions remain the compatibility fallback when external writes are disabled;
they never depend on an installed directory. Normal cloud qualification uses remote
object storage.

The initial create response, including its original version, can be replayed after
a lost response and subsequent edits. Its durable API receipt retains the response
projection without Markdown; the immutable version supplies the body. Receipts
remain scoped to company, actor and operation context. A deleted skill's receipt
fails explicitly rather than creating another identity.

`skill-execution-content.ts` owns temporary materialization. Heartbeat preparation
selects authorized immutable versions, saves the selection (including an empty
selection), and only then materializes admitted entries while retaining the existing
retention lock. A renamed skill is recovered through the version's stable skill ID.
Native recovery resolves the version, reconstructs its v1 projection and requires
the admitted digest, manifest digest, file count and byte count to match. Adjacent
compatible releases must preserve that projection contract.

Dynamic connector skills include run-specific resources and tool definitions.
They retain null version IDs because their bytes differ from the static company
skill. With S3 configured, native runtime selection stages and verifies their exact
packages before its admission transaction. That transaction checks the frozen input
and cancellation state, locks the package rows against collection, and publishes
the input and its preservation receipt together. Storage failure leaves the input
unpublished. Existing receipts retain their capture provenance on same-run recovery;
hot cache reads can continue during a storage outage. Local storage keeps its
existing volume-backed behavior.

Older native inputs can contain immutable skill bundles without a version ID.
Retained recovery obligations and current task sessions keep those bundles as
local durability dependencies until the `runtime-assets` maintenance phase copies
their authenticated files into the existing package store. The receipt binds the
original run input, skill selection, native manifest and package content hash.
Recovery restores the original bytes and read/execute modes, verifies the admitted
digests, and preserves the null version ID. Package collection protects these
receipts under the same row lock used by receipt publication.

This conversion may run online, including for an active run: it only copies an
immutable bundle under its publication/collection lock and appends an input-bound
receipt. It does not claim provider or controller ownership, settle recovery,
start a turn, change task intent or remove local data. Missing or changed sources
remain blockers. Ordinary new admission without a completed execution input is
not itself a local asset obligation. Both migration and replacement startup check
the concrete retained asset obligations independently of installation eligibility.

Run on the host retaining the original runtime asset files, and follow its cursor:

```sh
pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=runtime-assets --company=<uuid> --limit=10
pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=runtime-assets --company=<uuid> --limit=10 --apply
```

Studio tests deliver the selected saved version through the same native Runner
path, even without a permanent agent assignment. Their `skill_test` mode remains
explicit in the execution input. The durable test row replaces ordinary agent
assignments with its exact version, independently of the Beta skills preference;
required environment skills remain subject to their existing policy. A shadowed
or inaccessible test skill fails preparation. Preconfigured Runner homes cannot
receive this managed delivery, so Studio requires an isolated home. Task-bound
credentials and native authorization
retain the admitted scope through restart and task-mode edits. Native tools expose
only the harness task and admitted skills; company browsing, task creation,
connectors and assigned MCP tools are unavailable. A rollback image must understand
that mode to continue an active Studio test. Settle tests before rolling back to an
earlier image that rejects it.

Native task completion publishes the Studio result and output snapshot in the
same transaction as its status decision. Replaying a committed decision cannot
recapture later document edits or duplicate completion activity. A successful
provider turn alone does not complete a test whose task remains unfinished.

Bundled release snapshots remain immutable. Reconciliation preserves an installed
head; an older process must never promote its shipped bytes over another installed
version. The shipped release history remains independently selectable. Linked-local
skills retain mutable source behavior and are explicitly excluded from cloud readiness.

## One-time conversion

First deploy compatible readers and shared/API types. Preserve the old volume and
keep legacy writers until the final supported contents can be captured. Automatic
JSONB package migration is separate from installation conversion; passing one does
not prove the other is finished.

Run the existing bounded CLI with the target's explicit database and storage config:

```sh
pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=eligibility --company=<uuid> --limit=10
pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=installations --company=<uuid> --limit=10
```

These commands return a cursor. Continue each phase with `--after=<uuid>` until an
empty batch. Eligibility is a metadata-only report, not a storage verification claim.
Installation preview inspects candidate legacy contents without publishing them.
For remote legacy imports it can acquire their exact pinned source; it must never
use the latest branch as a replacement for missing installed content.

1. Inventory all companies and source kinds. Identify missing versions, incomplete
   inventory, local storage receipts, catalog reset snapshots and unresolved runs.
2. Backfill legacy package/version bodies with the existing `snapshots` and
   `versions` phases. The latter also fills missing version metadata indexes so
   subsequent library/history reads need no object downloads. Keep original bytes
   until independent verification succeeds.
3. Copy installations on the host with the legacy files. This phase shares the
   cache, publication and row locks with normal writers. It checks the exact
   local source again before publication:

   ```sh
   pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=installations --company=<uuid> --limit=10 --apply
   ```

4. Keep all original directories, versions and inline bytes. Conversion does not
   settle old execution, recovery or pin uncertainty. Preserve or settle every
   remaining local dependency through its owner before removing the volume.
5. Sweep `eligibility` and `verification`, including all historical versions. The
   latter verifies object bytes independently of caches. Follow all cursors; a
   successful first batch is not a complete readiness report.
6. Start a compatible server with a fresh home and test list, file read, authoring,
   history and exact selected execution before reopening normal admission.

Managed conversion preserves skill IDs, keys, history, assignments, pins and access
provenance. Identical complete snapshots reuse their version IDs. Catalog conversion
preserves the original reset version as well as installed content. A missing legacy
directory can use a complete verified saved version. A partial Markdown-only rescue,
unknown revision or unsafe source tree blocks conversion. An active legacy selection
without exact version identity remains a retention and volume-removal blocker even
after copy-only conversion. Preserve or settle such runs while A is still available;
do not cancel them simply to pass preflight.

Report statuses distinguish linked-local sources, legacy managed/catalog records,
missing current versions, unknown active-run selections, missing/corrupt packages,
local storage and provider mismatch. Expired unused Git history keeps its existing
30-day/exact-commit policy. Current, pinned, authored and recovering versions need
independent durable content. `local_storage` is supported self-hosted durability;
it does not satisfy remote empty-disk recovery.

## Storage and rollback

Cloud readiness requires external Postgres and remote object storage. Migrate and
verify existing local-disk objects before changing provider/bucket configuration;
changing configuration alone does not copy receipts or objects. Keep the old provider
available until a provider migration has copied and independently verified every
referenced object and atomically updated its storage receipt. This command does not
perform provider migration: `local_storage` and `provider_mismatch` are blocking
results for cloud qualification, not instructions to switch configuration and retry.
A directory of extracted caches is never a durable object backend.

The maintenance CLI reads the existing storage configuration directly so it can
verify legacy local-disk objects before cutover. On Cloud it still requires the
persisted runtime configuration and never seeds or changes that configuration.
This maintenance boundary does not start a server or relax Cloud server startup
requirements for external PostgreSQL, shared object storage and a stable secret
key. Apply operations retain their existing stopped-writer gates.

Conversion establishes the rollback floor. Afterward only compatible version-reading
binaries may overlap or be rolled back against the same remote state. Reattaching an
old volume does not roll back committed content. Filesystem-authoritative binaries
require an explicit reverse migration and are unsupported rollback targets.
A routine compatible A → B upgrade needs no skills conversion or filesystem copy.

A current version ID alone does not establish conversion completeness. Eligibility
and independent verification compare its file inventory with the installed record
and report `incomplete_current_version` when supporting files are missing. Installation
conversion can reacquire a complete exact pinned source; unavailable or unpinned
sources remain blockers. Historical versions are not compared with newer installed
inventories, since later revisions may legitimately add files.

## Qualification

The focused service suite covers publication rollback/conflict, supporting and binary
files, executable modes, restore/fork/rename, active-run deletion protection and legacy
conversion. The opt-in process suite starts independent Node servers with one database
connection each, exercises real HTTP routes and object storage, removes predecessor
homes and executes a delivered harmless script. It also injects a lost response, failed
upload and process death during upload. It is a skills-router qualification, not a
full hosted Runner or whole-application deployment claim.

Use owned disposable Postgres and S3-compatible fixtures. The optional admin URL
creates a random database per test suite and never migrates the database named in
that URL. The S3 test creates and removes its own random bucket. Supply test-only AWS
credentials through the environment:

```sh
PAPERCLIP_TEST_POSTGRES_ADMIN_URL=<disposable-postgres-url> \
PAPERCLIP_TEST_S3_ENDPOINT=<disposable-s3-endpoint> \
pnpm exec vitest run server/src/__tests__/ephemeral-skills-process.test.ts
```

Record revision, fixture image digests, version IDs, cache/object reads, cold/warm
timings and useful output. These local timings do not establish hosted latency.
Separately qualify one disposable staging tenant and a real remote Runner, including
an explicit repository import, historical pin, fresh-home replacements, compatible
rollback and recovery without predecessor paths. Publish the observed evidence with
the PR; a passing local suite is not that staging evidence.