## Thinking Path > - Paperclip manages work for AI agents. > - SSH environments run tasks on computers that users own. > - A Paperclip server needs network access before it can reach a private computer. > - A reusable Tailscale connection belongs in Connectors. > - Connected machines should become choices in the existing SSH environment form. > - This change connects a tailnet and resolves selected machines through a private network client. > - Agents retain their choice of harness and AI account. ## Linked Issues or Issue Description **Subsystem affected** Connectors, environment settings, SSH execution, company secret storage, and runtime packaging. **Problem or motivation** Entering a Tailscale address in the SSH form does not connect the Paperclip server to that tailnet. Requiring a separate device key for every environment also duplicates network setup. **Proposed solution** Add a Tailscale Connector with **Connect Tailscale Network**. Use brokered OAuth consent to enroll one private network client per connection. Let SSH environments select a connected network and stable machine identity. Preserve the enrolled node state through the company vault so a restart does not reuse the one-time token. **Alternatives considered** A separate Tailscale environment type would duplicate SSH behavior and mix network access with harness selection. This change uses the existing `ssh` driver. Related #15648 adds OAuth client credentials and a health check; this change uses user consent and supplies outbound task execution. Related #11743 concerns HTTPS brokering. **Roadmap alignment** This extends the existing Connected Apps, environment, and Personal & Shared AI Accounts capabilities. This PR is stacked on #21248, which provides SSH account selection independently of Tailscale. The hosted OAuth broker is a companion control-plane change. ## What Changed - Preserve Cloud reauthentication in the current browser tab. Explain canceled or failed consent in the connector and offer a retry of the same authorized saved connection. - Record Cloud enrollment starts initiated through the Tailscale connector in the company activity log. - Check install-preparation descendant liveness through the kernel process table in the Runner support test. Keep the direct child’s strict reaping assertion and reject any non-zombie descendant. - Align the application image stage contract with the Tailscale client stage after the upstream Runner-image refactor. Keep the exact stage list and check every application stage, including `tailscale-client`, for prohibited Runner compiler commands. - Group Tailscale consent, discovery, daemon lifecycle, pairing, and deletion under `server/src/services/connectors/tailscale/`, with their service tests. Update existing route, runtime, environment, vault, and company-deletion imports. Keep the existing contracts, authorization, audit, and transaction owners. - Handle early stdin closure in the Kubernetes real-shell test helper. Stop the input stream on `EPIPE`, preserve nonzero child exits, and reject a zero exit when input was lost. Add two real-process regression cases. Production transfer code and confinement checks are unchanged. - Read configured Tailscale executable paths when a client starts, after supported `.env` loading. Keep explicit command overrides available. - Discard pending pairing results after switching to Direct SSH or closing the form. Retain the existing company, connection, machine, and port checks. - Store network credentials with an immutable server-owned purpose from their first vault row. Exclude them from generic catalogs, bindings, environment projection, and value reads. Use a dedicated audited server consumer for pinned connector reads. Block unclassified legacy serialized node state at the same value boundary. Keep SSH private-key binding separate from connector credentials. - Gate network queries and setup on verified local-implicit or instance-admin access. Hide Tailscale catalog rows for other members, show an admin requirement on direct setup navigation, and avoid device discovery before management access is known. - Keep failed-client cleanup in the network manager. A failed request cannot disconnect a newer retry for the same identity; explicit connector disconnect and enrollment retirement still stop their networks. - Check for readable Ed25519, ECDSA, or RSA host public keys before pairing edits `authorized_keys`. Print a known-hosts entry for every available key, or fail before changing authorized keys if none are available. - Return self-hosted Cloud enrollment to the Tailscale connector. Save the unprefixed setup path and let the shared callback add the verified company prefix. Keep external and unrelated destinations rejected. - Hide pairing instructions after an SSH port change. Ignore late key-selection results for the old port. Operators prepare again for matching known-hosts output. - Retire connector references before deleting a company or its vault credential. Stop company daemons after the database commit. Keep failed secret retirement retryable and prevent late node-state writes from restoring deleted connections. Discard retired state without blocking behind the vault deletion lock. - Reload a mismatched credential before replacing a ready daemon. Recheck the active entry and manager generation after the read. An older request cannot stop a newer healthy socket or cross shutdown. - Pin connector credentials to the exact vault version and SHA-256 digest recorded by trusted enrollment and node-state writes. Ignore generic secret rotations. Advance both fields with a compare-and-set guard. Reject unpinned or tampered credentials before network startup. - Save an explicit Direct SSH selection and clear old Tailscale fields after config merging. Preserve the operator's entered direct host. - Route all Tailscale SSH runtime access through the connection service after SSH-key loading. Pin access to the environment's saved connector identity and recheck after device discovery. Remove standalone key-based daemon startup and reject raw enrollment keys or credential-only network configurations. Align the shared config type and SDK pins. - Correct two E2E assumptions exposed during connector qualification. Keep the two-window chat fixture stale until its own mutation. Validate the original failed run and its permitted write-revocation timestamp in the legacy recovery browser test. - Add the Tailscale Connector page, catalog entry, OAuth return flow, connected-network management, device discovery, and disconnect. Document all six routes in OpenAPI and the API reference, including their instance-admin requirement. - Add company-scoped connection storage in `0480_lazy_lethal_legion`, trusted credential version/digest fields in `0481_round_pet_avengers`, and server-owned secret purpose/backfill in `0482_aberrant_rumiko_fujikawa`. Generate all three migrations through the maintained command and retain the five newest snapshots. - Add machine pairing that generates a fresh SSH key per request, stores its private half in the company vault, and uses the generated public half in the install command. Do not trust editable saved metadata. Preserve existing authorized keys, including a file without a final newline. Keep SSH username, workspace, and trusted host keys explicit. - Use the existing SSH driver. Resolve the machine's current address from its stable device ID. Reject disconnected, removed, offline, and wrong-company selections. - Hide Network when Direct is the only option. Select a sole connected tailnet automatically and hide its selector. Remove the requested setup helper copy. Do not restrict the harness to Claude. - Run a private userspace Tailscale daemon with a separate socket. Persist its node state before execution. Restore it after restart, bound startup, serialize reconnect cleanup, and close it during server shutdown. - Guard OAuth completion with a durable attempt marker. Disconnect and newer authorization invalidate older callbacks. Close replaced daemons and remove cancelled enrollment secrets without changing newer connection state. - Preserve SSH host verification and tailnet-specific session identity. Honor the configured Tailscale executable path, including spaces and OpenSSH percent escaping. - Package checksum-pinned Tailscale clients and their license in the standard runtime image. Exercise the client build stage in PR CI and include it in image smoke checks. ## Verification - Merge status: all CI and image checks pass, and current-head Apex is 5/5 with no unresolved comments. GitHub rejected a direct merge because the active master ruleset requires code-owner approval for `.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3) applies; eligible maintainers already have review requests. Auto-merge is enabled with the squash method for the verified head. The PR remains open pending that required approval; no branch policy was bypassed. - Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI verification passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320). All required checks pass, all applicable test/build gates are green, and the branch is mergeable against master. SSH prerequisite #21248 and Cloud broker paperclipai/paperclip-cloud#1314 are already merged. - [Apex reviewed this exact revision at 5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437). There are no unresolved review threads. The PR is classified as Critical impact. - Final image qualification passes for the [application Runner path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267), [amd64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198), and [arm64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156). Typechecks, Runner support, native/Rust tests, provider tests, serialized-server suites, browser shards, and the remaining applicable CI checks pass. This is source/image CI qualification, not a production rollout. - The last master conflict affected only Docker stage declarations and their boundary test. Both the checksum-pinned Tailscale stage and upstream `runner-lockfile` guard are retained, including the upstream `deps` inheritance. The Tailscale client stage is byte-identical. All 180 cases across five Docker/image contract suites pass locally, along with server and UI typechecks. The maintained 543-input SDK check has no drift. Runtime and lock pins are unchanged. Secret/PII and whitespace scans pass. - Seven OAuth regression cases failed before the fixes. The current-tab test uses the real handoff helper, checks that the session is stored before reauthentication navigation, mounts the shared return page, resumes consent, and verifies handoff cleanup. Callback tests check trusted saved-connection context, fixed safe outcomes, failure audit metadata, denied membership and expired-state rejection, JSON API errors, and no provider text in browser returns. UI retries require a connection in the current company's catalog. All 80 OAuth route/service/UI/shared-handoff cases pass; server and UI source/test typechecks pass. - [Server shard 4/12 on the prior refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436) passed 2,784 cases and skipped 21, then failed a reassignment fixture's `TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its automatic retry found the surviving `REA` prefix. The unchanged test passes all three cases locally. A controlled reader/cleanup overlap reproduces the same deadlock, but the exact competing CI query was not logged. Each case now has a distinct company prefix and identifier; the owned database is removed at suite end. The three test bodies and assertions are byte-identical. Eight local reassignment/lock-release cases pass, and the [final server shard 4/12 passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392). Production reassignment logic, deadlines, and retries are unchanged. - The initial master integration passed 330 cases across 17 suites. Coverage includes connector state, device discovery, network lifecycle, pairing, routes, SSH binding and exports, environment persistence, and connector UI. All three unpublished migrations were regenerated after master's `0479_short_darkstar`: `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL body is byte-identical to its earlier unpublished migration. Generated snapshots preserve existing master tables and add only the intended Tailscale table and secret-purpose column. Exactly five snapshots remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift checks pass. The later Docker refresh did not alter this migration sequence. - Earlier attempts remain part of the verification record. On `1c020a0`, Rust cache restoration and server shard 5/12 were interrupted by shutdown signals, Runner shard 2/3 lost communication with no retrievable log, and an amd64 provider download failed with npm `ECONNRESET`. Same-head retries recovered, without establishing the cause of the communication loss. Later superseded runs also had cancellations with incomplete logs. These interrupted attempts are not counted as passes. The final revision has fresh successful CI. - Local platform limits remain recorded. The Kubernetes suite on macOS passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and skipped four Linux-only cases. Unchanged source reproduced the same 12 failures. An initial run also lacked provider dependencies until the maintained preparation command installed them. Those local attempts are not green qualification. Final Linux provider CI passes. Local Docker qualification was unavailable; final image checks passed in CI. - Before PR separation, a live tailnet/SSH Claude Code task completed on Ubuntu ARM64 without provider login for the remote user. The selected Paperclip subscription ran it, wrote the expected file, posted completion, and marked the task done. File contents and private runtime cleanup were checked. This is prior live evidence; the final revision was not live-retested. Other providers have automated coverage only. Hosted registration for unrelated tailnets and production concurrency remain qualification limits below. - The Superagent security and supply-chain reports remain neutral and inconclusive. Earlier attempts to restart the failed security scan returned HTTP 404. Supply-chain intelligence lacked coverage for extracted pinned dependencies and the shared workflow reference. These reports do not establish successful security qualification. The required CI and review gates pass; this external coverage gap remains disclosed. ## Risks - The Superagent security and supply-chain reports are inconclusive. The security scan re-run request returned HTTP 404, and the supply-chain scanner lacks coverage for the extracted dependencies and shared workflow reference. These neutral results are not successful security qualification. - The server-owned secret purpose is internal database metadata, not an HTTP-writable field. The generic secret service owns enforcement so direct API reads and runtime projection share the same restriction. Previously copied node state remains valid outside Paperclip until the enrolled device is revoked in Tailscale; a local disconnect cannot stop an external copy. - Tailscale network configurations now require a connected Connector and selected machine. The unreleased key-only path is removed; any saved configuration from that path needs Connector setup. - The connector remains a pilot. The local test used an OAuth app registered in the consenting user's tailnet. Qualify hosted app registration for unrelated customer tailnets before general availability. - This implementation supports one serving process per instance. Concurrent replicas need exclusive ownership of the persisted network identity before production rollout. Preserve the database and company vault master key across restarts. - Each machine-pairing preparation creates a new vault secret. Unused generated keys follow the existing secret deletion lifecycle. A saved SSH key remains available as an explicit choice. - Apply additive migrations `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before application rollout. The new credential version and digest columns are nullable. Existing unpinned connections require fresh OAuth consent; the migration deliberately does not trust or backfill a generic secret value. Regenerate or renumber migrations if master advances the sequence before merge. - The target needs an SSH server, its chosen harness, and allowed tailnet access. A reachable network does not grant SSH or provider authorization. Pairing reads standard `/etc/ssh` host-public-key files; custom host-key locations require manual known-hosts configuration. - Roll back the application revision to remove new entry points. Existing connection secrets and registered Tailscale nodes retain their separate lifecycle. Do not drop the table or delete node state as part of an application rollback. - Production qualification still needs denied-policy, revocation, cancellation, and deployment-overlap checks. Local Docker and a production deployment were not performed. ## Model Used OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and this extends existing connector and environment features - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the feature request fields - [x] I have not referenced internal issues, local URLs, or private machine identifiers - [x] My branch name describes the change - [x] I have run focused tests locally and they pass - [x] I have added tests for new behavior and failure paths - [x] I have updated documentation - [x] I have documented migration and runtime risks - [x] All Paperclip CI gates are green - [ ] Required code-owner approval received for the workflow change - [x] Greptile Apex is 5/5 on the final head with no unresolved comments - [x] I will address CI and review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
@paperclipai/plugin-kubernetes (alpha)
First-party Paperclip sandbox-provider plugin for Kubernetes.
Alpha: the default backend (sandbox-cr) is built on kubernetes-sigs/agent-sandbox v1alpha1 — expect breaking changes as that CRD evolves toward Beta. A stable fallback backend (job, using batch/v1 Job) is available for clusters without agent-sandbox installed, but it does NOT support multi-command exec (paperclip-server's adapter-install pattern requires sandbox-cr).
Prerequisites
For sandbox-cr backend (default, recommended)
- A Kubernetes cluster running k8s 1.27+
kubernetes-sigs/agent-sandboxcontroller installed in the cluster (alpha — installs thesandboxes.agents.x-k8s.io/v1alpha1CRD and controller)- Paperclip-server running with access to the cluster (in-cluster via
inCluster: trueor external viakubeconfig)
For job backend (stable fallback)
- A Kubernetes cluster running k8s 1.27+
- Paperclip-server with cluster access — no additional controllers or CRDs required
Installation
paperclipai plugin install @paperclipai/plugin-kubernetes
Or, for local development:
paperclipai plugin install --local /path/to/paperclip/packages/plugins/sandbox-providers/kubernetes
Backends
The plugin supports two backend modes, selected via the backend config field:
| Backend | Default | Stability | Multi-command exec | Requires |
|---|---|---|---|---|
sandbox-cr |
Yes | Alpha | Yes | kubernetes-sigs/agent-sandbox controller |
job |
No | Stable | No | Nothing beyond k8s 1.27+ |
sandbox-cr (default): Creates a Sandbox CR (agents.x-k8s.io/v1alpha1) whose controller provisions a long-lived pod running sleep infinity. paperclip-server execs individual commands into the running pod — this is the multi-command adapter-install pattern. When you releaseLease, the Sandbox CR is deleted and the controller tears down the pod.
job (stable fallback): Creates a batch/v1 Job. The container entrypoint runs once and exits — no multi-command exec possible. Use this when you cannot install agent-sandbox, or when you need strictly stable Kubernetes APIs. Note: paperclip-server's adapter-install pattern will not work in job mode.
Migrating from job to sandbox-cr
- Install the agent-sandbox controller:
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/sandbox.yaml - Update your environment config to set
backend: "sandbox-cr"(or removebackendsincesandbox-cris the default) - New leases will use the Sandbox CR backend. Existing leases created with
jobmode continue to use job semantics until they are released.
Configuration
Create a sandbox environment with driver: kubernetes. One of these auth fields is required:
inCluster: true— use the in-pod ServiceAccount credentials (when paperclip-server runs inside the same cluster).kubeconfig: <YAML>— inline kubeconfig (stored as a company secret).kubeconfigSecretRef: <secret-uuid>— reference to an existing Paperclip secret.
Common optional fields:
| Field | Default | Purpose |
|---|---|---|
backend |
"sandbox-cr" |
sandbox-cr (alpha, requires agent-sandbox controller) or job (stable, one-shot entrypoint). |
adapterType |
"claude_local" |
One of the supported adapter types (claude_local, codex_local, gemini_local, cursor_local, opencode_local, pi_local). Determines runtime image + env keys + egress allow-list. |
namespacePrefix |
"paperclip-" |
Prefix for the per-company tenant namespace. |
companySlug |
derived from companyId | Override the auto-derived company slug. |
imageRegistry |
(none) | Override the default registry for agent runtime images. |
imageAllowList |
[] |
Glob patterns of allowed target.imageOverride values. Empty = no override permitted. |
imagePullSecrets |
[] |
Names of pre-created Docker image pull secrets in the tenant namespace. |
egressAllowFqdns |
[] |
Additional FQDNs (beyond adapter defaults like api.anthropic.com). |
egressAllowCidrs |
[] |
Additional CIDRs to allow egress to. |
egressMode |
"standard" |
standard (NetworkPolicy + CIDRs) or cilium (CiliumNetworkPolicy + FQDN allow-list). |
runtimeClassName |
(none) | e.g. kata-fc for Firecracker-backed microVMs. Cluster must have the RuntimeClass installed. |
serviceAccountAnnotations |
{} |
Annotations applied to per-tenant ServiceAccount (e.g. IRSA eks.amazonaws.com/role-arn). |
jobTtlSecondsAfterFinished |
900 |
Seconds after a Job completes before garbage-collection. |
podActivityDeadlineSec |
3600 |
Hard ceiling on a single run's wall-clock time. |
Full JSON Schema in src/manifest.ts.
Task-scoped egress grants
Keep provider-level egress defaults narrow, then grant only the destinations a task needs through its execution workspace settings:
{
"executionWorkspaceSettings": {
"networkEgress": {
"allowFqdns": ["github.com", "pypi.org"],
"allowCidrs": []
}
}
}
The provider creates a workload-owned policy selected by the task run label, so the additional destinations do not become reachable from other concurrent agent pods. Cilium mode enforces FQDNs directly. Standard NetworkPolicy mode cannot express FQDNs, so an FQDN grant permits public IPv4 TCP 80/443 for that run while excluding private, loopback, link-local, CGNAT, and multicast ranges. Network failures that look policy-related include the grant path in stderr, and the sandbox exposes the effective policy through PAPERCLIP_NETWORK_EGRESS_* environment variables.
What gets created in your cluster
For each company that runs agents (created lazily on first dispatch):
Namespace paperclip-{companySlug} (PSS: restricted enforce + audit)
ServiceAccount paperclip-tenant-sa
Role paperclip-tenant-role (only get pods/log)
RoleBinding paperclip-tenant-rb
ResourceQuota paperclip-quota (pods, requests/limits cpu+memory)
LimitRange paperclip-limits (container max/min/default/defaultRequest)
NetworkPolicy paperclip-deny-all (deny ingress + egress baseline)
NetworkPolicy paperclip-egress-allow (DNS + paperclip-server callback + user CIDRs)
OR CiliumNetworkPolicy paperclip-egress-fqdn if egressMode=cilium
For each agent run (sandbox-cr backend):
Sandbox CR pc-{ulid} (agents.x-k8s.io/v1alpha1; explicit delete on release)
Pod pc-{ulid}-{podSuffix} (managed by Sandbox controller; torn down on CR delete)
Secret pc-{ulid}-env (owned by Sandbox CR; cascade-deleted)
For each agent run (job backend):
Job pc-{ulid} (backoffLimit: 0, ttlSecondsAfterFinished from config)
Pod pc-{ulid}-{podSuffix} (owned by Job; cascade-deleted)
Secret pc-{ulid}-env (owned by Job; cascade-deleted)
Security baseline
Every agent pod is:
- non-root (
runAsUser: 1000,runAsGroup: 1000,runAsNonRoot: true) - drops ALL Linux capabilities,
allowPrivilegeEscalation: false readOnlyRootFilesystem: truewith explicitemptyDirmounts for/workspace,/home/paperclip,/home/paperclip/.cache,/tmpseccompProfile: RuntimeDefault- Tini as PID 1 (reaps zombies, forwards signals)
fsGroupChangePolicy: OnRootMismatch(fast PVC startup; openclaw-operator lesson)automountServiceAccountToken: true(for the agent shim's paperclip-server callback)
Plus per-namespace pod-security.kubernetes.io/enforce: restricted and a deny-all NetworkPolicy baseline with explicit egress allow-list (DNS, paperclip-server, configured FQDNs/CIDRs).
The per-run Secret carrying the bootstrap token and adapter API keys has ownerReferences pointing at the owning Job, so a single kubectl delete job … cascades cleanly to the Pod and Secret.
Optional Kata-FC microVM isolation
For stronger isolation, install Kata Containers with the Firecracker hypervisor, then set runtimeClassName: kata-fc in the plugin config. Each agent pod will run inside a Firecracker microVM. Requires nested-virt-capable nodes (bare-metal or specific cloud instance types).
Roadmap
- Phase A (done):
sandbox-crbackend — multi-command exec via agent-sandbox Sandbox CRD. - Phase B: Warm pool support — pre-provisioned Sandbox CRs for sub-second cold starts. The
SandboxOrchestratorinterface reserves optionalpause?/resume?extension slots. - Phase C: Kata-FC + snapshots —
runtimeClassName: kata-fcwith VM snapshot for fast restore. - Phase D: Contribute back to agent-sandbox upstream if their Beta model diverges from our needs. The
SandboxOrchestratorinterface (src/sandbox-orchestrator.ts) is the clean swap point — a new implementation can be added without touchingplugin.tsbusiness logic.
Lessons learned (from openclaw-operator)
This plugin adopts patterns from openclaw-rocks/openclaw-operator:
- Tini PID 1 (issue #471 — zombie helper processes)
- Read-only rootFS with explicit writable mounts (issue #456 — ~/.config not writable)
- Strategic merge on reconcile (issue #446 — preserve third-party annotations)
- Multi-storage-class testing (issue #448 —
local-path-provisionerdifferences) - Image version compat matrix (issue #462 — runtime deps cannot resolve after upgrade)
Development
cd packages/plugins/sandbox-providers/kubernetes
pnpm install --ignore-workspace
pnpm test # unit tests only (fast)
pnpm typecheck
pnpm build
To run the kind-cluster integration test (requires kubectl --context kind-paperclip and a pre-loaded alpine image; see test/integration/end-to-end-run.test.ts):
RUN_K8S_INTEGRATION_TESTS=1 pnpm test test/integration/end-to-end-run.test.ts