PaperClipAI/packages/plugins/sandbox-providers/exe-dev/README.md

# Experimental durable exe.dev environments

Enable **exe.dev Environments** in Instance → Experimental, install this bundled
plugin, and create a sandbox environment using **exe.dev VM (experimental)**.
Daytona remains the preferred Cloud default. Cloud installs this plugin only for
releases and stacks explicitly opted into `enableExeEnvironments` with `exe-dev`
selected in the release-validated bundled plugin list.

An environment owns one named VM. Each agent/workspace lease has a separate home,
working directory, socket, and systemd cgroup on that VM. Agents sharing a VM must
trust each other: these directories are not security boundaries. A VM cannot be
bound to two Paperclip companies or environments.

## Credentials and creation

Register an SSH public key with exe.dev; save its matching private key in a
Paperclip company secret using the environment form. Management commands and VM
execution both use SSH, so an exe.dev HTTPS API token is unnecessary. An operator
may instead configure an absolute `sshIdentityFile`, or use
`EXE_DEV_SSH_PRIVATE_KEY` / `EXE_DEV_SSH_KEY_FILE` on a self-hosted controller.
Credentials remain on the controller and are never forwarded to the VM.
For a private image registry, set `registryAuth` to a company secret containing
`username:token` (or `EXE_DEV_REGISTRY_AUTH` for the explicit live harness). exe.dev
uses this credential to pull the image; agent commands do not receive it.

- **Attach:** specify `vmName` for an existing compatible VM.
- **Create:** specify an immutable `image@sha256:…` published from
  `docker/exe-dev-runner/Dockerfile`. A stable name is derived from the company and
  environment unless `vmName` is supplied. Retrying an uncertain create reconciles
  that name. A connection test validates the account; the first run provisions
  and verifies the image. An attached VM is inspected without modification.

Arbitrary images and boot-time package installation are unsupported. The image
contains Node, systemd, the execution supervisor, the Rust runner, its matching
provider pack, and the supported legacy CLIs. Missing runtimes fail with an image
compatibility error. Existing 0.1 per-run configurations require explicit migration
to a compatible named VM; upgrading the plugin never deletes those old VMs.

SSH uses a dedicated identity, disables agent forwarding, and rejects changed
host keys. `accept-new` persists first-use keys in
`~/.ssh/paperclip-exe-known_hosts`. For preverified keys, supply `knownHosts` entries
for both `exe.dev` and the VM; this forces strict checking.

## Lifetime and concurrency

Normal release retains the VM and reusable lease. Cancellation or lease destruction
stops only that lease's cgroup, waits for confirmation, and leaves files intact.
An SSH disconnection is not a termination receipt. A requested lease deadline
limits its systemd service; it never deletes the VM. Controller restart reattaches
to the same VM identity and disk. Missing or replaced VMs fail closed and require
explicit recovery into a new environment.

The controller stores the resource identity in reserved environment metadata,
independently of run leases. Concurrent acquisitions must agree on that identity.
A durable marker on the VM detects replacement even if its DNS name is reused.
There is no automatic VM deletion, disk reclamation, or in-place image upgrade.
Disconnecting an environment retains its VM; delete it explicitly in exe.dev when
its data is no longer needed.

Reusable directories are scoped to the agent and project workspace, or to the
agent and task when no project workspace exists. Concurrent runs receive distinct
leases; a retained lease can be reused only after its previous run releases it.

Agent homes and workspaces live under `/var/lib/paperclip-exe/<scope>/<lease>/`.
Use the normal isolated-workspace policy for concurrent task work. Separate Git
worktrees avoid two agents modifying the same checkout. This feature does not add
a service registry or port allocator. Start a persistent web server as a separate
systemd service if it must survive agent cancellation; choose its port explicitly.

VM previews must remain private and use exe.dev's own HTTPS sharing/authentication.
Native runners connect outward to a reachable `PAPERCLIP_RUNNER_PUBLIC_URL=wss://…`.
Cloud derives this origin from the stack's canonical hostname. Paperclip does not
proxy exe.dev preview pages.

## Files and backups

The first run seeds the remote workspace. Later runs adopt its persisted files;
legacy initialization is marked only after a successful seed. Completion uses the
existing baseline-based Git/file copyback pipeline. Unrelated host edits survive,
but a changed remote file takes precedence when the same file also changed on the
host. Use isolated worktrees for concurrent editing; this is not a conflict merge.
Native runs additionally use
the existing verified harness/session checkpoint and restart-recovery contract.
OpenCode checkpoints preserve its session database and history while excluding
disposable launch homes (including npm caches),
regenerated configuration (including launch credentials and npm executable
aliases) and caches. Codex launch credentials and scratch aliases are also excluded.
The existing exclusions still apply, including dependencies, generated/cache
folders, ignored files, and runtime scratch.

This is the same backup coverage as Daytona, not a disk image or continuous mirror
into object storage. In Cloud, copyback and native checkpoints under `PAPERCLIP_HOME`
land on the tenant's persistent volume, which has daily/weekly/monthly provider
backups. Explicitly uploaded artifacts use the separate object-storage path.
Files outside copied paths, live process memory, database consistency, ignored
files, and VM-wide state are not included. A controller filesystem outside the
backed-up volume needs its own backup policy. Restore is explicit and must be tested
against the data classes the application actually needs.

## Build and verification

```sh
pnpm --dir packages/plugins/sandbox-providers/exe-dev install --ignore-workspace
pnpm --dir packages/plugins/sandbox-providers/exe-dev build
pnpm --dir packages/plugins/sandbox-providers/exe-dev test
node cli/node_modules/tsx/dist/cli.mjs scripts/build-exe-dev-image.ts --push --metadata-file /tmp/exe-image.json
```

The manual **Experimental exe.dev image** workflow publishes a content-addressed
Linux amd64 image and an immutable digest artifact. Production builds require a
clean checkout and frozen dependencies. `PAPERCLIP_IMAGE_ALLOW_DIRTY=1` is only for
explicit local development builds.

Run the explicit paid browser matrix and provider stress campaign:

```sh
pnpm test:e2e:runner -- --suite exe-compatibility
pnpm test:e2e:runner -- --suite exe-recovery
pnpm test:e2e:runner -- --suite exe-warm-continuity
EXE_DEV_LIVE_SMOKE=1 node cli/node_modules/tsx/dist/cli.mjs tests/exe-dev-live/stress.ts
```

Both require `EXE_DEV_SSH_PRIVATE_KEY` and `PAPERCLIP_E2E_EXE_IMAGE`; browser cells
also require their model credentials. Native browser cells need an externally
reachable WSS origin. Native OpenCode/ACPX also require
`PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` pointing to a build-owned Linux amd64
provider pack on the controller. The Cloud image builds and configures this pack;
self-hosted controllers must export `/opt/paperclip-runner/provider-pack` from the
matching exe image and configure its local path (including on macOS). Never use a
macOS provider pack for a Linux VM. For local testing, set `PAPERCLIP_E2E_RUNNER_TUNNEL_BIN` to an
installed `cloudflared` binary. The harness exposes only authenticated runner
WebSocket upgrades; all ordinary HTTP requests and board API paths return 404.
The matrix has seven profiles × three workflows. Stress
covers eight shared leases, concurrent commands, UTF-8/stdin, copyback, restart,
SSH loss, VM reboot, large output, escaped descendant cancellation, timeout,
cancellation isolation, and a separate private HTTP service. Set
`EXE_DEV_SOAK_MS` for an explicit idle soak. Campaign cleanup deletes only
campaign-owned VMs; production provider hooks never delete VMs.

The native Codex message cell additionally runs two native and one legacy agent
concurrently on its VM and checks distinct homes/workspaces. The four-cell
recovery suite covers structured question/resume with and without a controller
restart for legacy and native Codex. The two-cell warm suite checks three turns
on one workspace and stable native runner/process/session identity.
For longer campaigns, accountless Cloudflare tunnels can rate-limit new endpoints.
Use a single operator-owned WSS relay instead: run `tests/exe-dev-live/relay.ts`
with `PAPERCLIP_E2E_RUNNER_RELAY_REGISTRY` set to a private local directory, and
forward its loopback listener through a TLS endpoint. Set the same registry path
and `PAPERCLIP_E2E_RUNNER_RELAY_URL=wss://your-relay.example` in the E2E process.
Each harness registers its own random route and removes it on shutdown. The relay
forwards only registered runner WebSocket upgrades, retains the runner's capability
authentication, and returns 404 for ordinary HTTP and board APIs. Its TLS/SSH
endpoint is test infrastructure and must be removed after qualification.

`tests/exe-dev-live/preview.ts` provides an
interactive private Vite fixture with `create`, `cancel`, `edit`, and `cleanup`
commands; it deliberately keeps its VM until cleanup so a signed-in browser can
verify HMR and an idle soak.