mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 02:32:07 +01:00
feat: add kubernetes execution target provisioning
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed
1 parent
0808b388ee
commit
4a7a45f960
87 files changed
+15005
-131
No files matched your search
@@ -0,0 +1,190 @@
|
||||
---
|
||||
title: Kubernetes Execution — Cluster RBAC
|
||||
summary: Reference ClusterRole for the Paperclip driver, per-rule rationale, and ServiceAccount binding templates for in-cluster and cross-cluster topologies
|
||||
---
|
||||
|
||||
The Paperclip driver needs a `ClusterRole` to provision and manage tenant namespaces across the cluster. This document explains every permission in that role, and provides binding templates for both supported topologies.
|
||||
|
||||
## Reference ClusterRole
|
||||
|
||||
The canonical source is `packages/adapters/kubernetes-execution/manifests/paperclip-tenant-manager-clusterrole.yaml`. Its contents are reproduced here for reference:
|
||||
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: paperclip-tenant-manager
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["namespaces", "resourcequotas", "limitranges", "secrets", "serviceaccounts", "configmaps", "persistentvolumeclaims", "pods", "pods/log", "pods/exec"]
|
||||
verbs: ["get", "list", "create", "update", "patch", "delete", "watch"]
|
||||
- apiGroups: ["batch"]
|
||||
resources: ["jobs"]
|
||||
verbs: ["get", "list", "create", "update", "patch", "delete", "watch"]
|
||||
- apiGroups: ["networking.k8s.io"]
|
||||
resources: ["networkpolicies"]
|
||||
verbs: ["get", "list", "create", "update", "patch", "delete"]
|
||||
- apiGroups: ["rbac.authorization.k8s.io"]
|
||||
resources: ["roles", "rolebindings"]
|
||||
verbs: ["get", "list", "create", "update", "patch", "delete"]
|
||||
- apiGroups: ["cilium.io"]
|
||||
resources: ["ciliumnetworkpolicies"]
|
||||
verbs: ["get", "list", "create", "update", "patch", "delete"]
|
||||
```
|
||||
|
||||
Apply it to a cluster with:
|
||||
|
||||
```bash
|
||||
kubectl apply -f packages/adapters/kubernetes-execution/manifests/paperclip-tenant-manager-clusterrole.yaml
|
||||
```
|
||||
|
||||
## Rule-by-rule rationale
|
||||
|
||||
### Core API group (`apiGroups: [""]`)
|
||||
|
||||
| Resource | Why the driver needs it |
|
||||
|----------|------------------------|
|
||||
| `namespaces` | Create and patch tenant namespaces (`paperclip-{companySlug}`). The driver reads the existing namespace to check for `paperclip.ai/managed-by=paperclip` before mutating — it will not touch namespaces it did not create. `watch` is used in M2 to track namespace deletion events. |
|
||||
| `resourcequotas` | Apply `paperclip-tenant-quota` to each namespace. Must `patch` on re-provision (quota overrides from `cluster_tenant_policies`). |
|
||||
| `limitranges` | Apply `paperclip-tenant-limits`. Must `patch` on billing tier changes. |
|
||||
| `secrets` | (a) Create `paperclip-image-pull` (registry credentials per namespace). (b) In M2: create per-Job ephemeral Secrets that hold resolved `secret_ref` values; these carry an `OwnerReference` to the Job so Kubernetes GCs them automatically. |
|
||||
| `serviceaccounts` | Create the `paperclip-agent` ServiceAccount in each namespace (`automountServiceAccountToken: false`). |
|
||||
| `configmaps` | Reserved for M2: workspace-init config and adapter configuration injection via `ConfigMap` volumes. |
|
||||
| `persistentvolumeclaims` | M2: create per-agent `PVC` (`agent-{agentSlug}-workspace`) for warm workspace storage. `watch` is needed to wait for the PVC to be bound before submitting the Job. |
|
||||
| `pods` | M2: read pod status to determine Job phase; gate on pod `Ready` condition. |
|
||||
| `pods/log` | M2: stream container logs from agent pods back to the Paperclip run log via `ctx.onLog`. |
|
||||
| `pods/exec` | M2: reserved for workspace-init debug flows and operator diagnostics. Not used in M1. |
|
||||
|
||||
### Batch API group (`apiGroups: ["batch"]`)
|
||||
|
||||
| Resource | Why the driver needs it |
|
||||
|----------|------------------------|
|
||||
| `jobs` | M2: submit one `Job` per agent run. The driver needs `watch` to track the Job to completion and map terminal conditions to `AdapterExecutionResult` exit codes. `delete` is used for cancellation. |
|
||||
|
||||
This rule is present in the ClusterRole now so that M2 can begin scheduling runs without an RBAC update. In M1, the driver's `run()` method returns `NOT_YET_SUPPORTED` before any Job is submitted.
|
||||
|
||||
### Networking API group (`apiGroups: ["networking.k8s.io"]`)
|
||||
|
||||
| Resource | Why the driver needs it |
|
||||
|----------|------------------------|
|
||||
| `networkpolicies` | Apply three `NetworkPolicy` objects per namespace: `default-deny-ingress`, `default-deny-egress`, `paperclip-agent-egress`. Must `patch` on re-provision to update the FQDN allowlist or control-plane selector. `watch` is not required (policies are reconciled at `ensure-tenant` time). |
|
||||
|
||||
### RBAC API group (`apiGroups: ["rbac.authorization.k8s.io"]`)
|
||||
|
||||
| Resource | Why the driver needs it |
|
||||
|----------|------------------------|
|
||||
| `roles` | Reserved for M2: in-namespace `Role` objects for fine-grained per-job access control. |
|
||||
| `rolebindings` | The driver creates a `RoleBinding` named `paperclip-driver` in each tenant namespace, binding the `paperclip-tenant-manager` ClusterRole to the driver ServiceAccount scoped to that namespace. This is required for the driver to have write access to namespaced resources after initial namespace creation. |
|
||||
|
||||
**Important:** The ability to create `RoleBindings` is a powerful permission. The driver is constrained to bind only the roles it already holds (the k8s API enforces that you cannot grant more than you have). The `paperclip-driver` identity should be protected: the ServiceAccount token must not be exposed outside the control plane.
|
||||
|
||||
### Cilium API group (`apiGroups: ["cilium.io"]`)
|
||||
|
||||
| Resource | Why the driver needs it |
|
||||
|----------|------------------------|
|
||||
| `ciliumnetworkpolicies` | When the cluster has the Cilium CNI, apply a `CiliumNetworkPolicy` alongside the vanilla NetworkPolicy. The CNP provides L7/FQDN egress filtering (see [security-model.md](./security-model.md#cilium-variant-fqdn-allowlist-auto-detected)). If Cilium is not present, this rule is harmless — the `cilium.io` API group simply does not exist. |
|
||||
|
||||
## Binding templates
|
||||
|
||||
### In-cluster topology
|
||||
|
||||
Paperclip server and agent workloads share the same cluster. The server pod's ServiceAccount in `paperclip-system` holds the driver identity.
|
||||
|
||||
```yaml
|
||||
# Step 1: namespace for the Paperclip control plane
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: paperclip-system
|
||||
---
|
||||
# Step 2: ServiceAccount for the driver
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: paperclip-driver
|
||||
namespace: paperclip-system
|
||||
---
|
||||
# Step 3: bind the ClusterRole cluster-wide
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: paperclip-driver
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: paperclip-driver
|
||||
namespace: paperclip-system
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: paperclip-tenant-manager
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
Apply:
|
||||
|
||||
```bash
|
||||
kubectl apply -f paperclip-system-sa.yaml
|
||||
```
|
||||
|
||||
The Paperclip server pod must run with `serviceAccountName: paperclip-driver`. The `@kubernetes/client-node` library will automatically use the in-cluster ServiceAccount token from `/var/run/secrets/kubernetes.io/serviceaccount/`.
|
||||
|
||||
When adding the cluster connection, use `--kind in-cluster` (no `--kubeconfig-secret`):
|
||||
|
||||
```bash
|
||||
paperclipai cluster add \
|
||||
--label production \
|
||||
--kind in-cluster
|
||||
```
|
||||
|
||||
### Cross-cluster topology
|
||||
|
||||
The Paperclip server runs outside the workload cluster. Access is via a stored kubeconfig whose embedded user must have the same permissions as the `paperclip-tenant-manager` ClusterRole.
|
||||
|
||||
The kubeconfig user identity is cluster-specific. Apply the ClusterRole and a `ClusterRoleBinding` on the **workload cluster** that binds the kubeconfig user:
|
||||
|
||||
```yaml
|
||||
# Apply to the workload cluster (not the control-plane cluster)
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: paperclip-driver-external
|
||||
subjects:
|
||||
- kind: User
|
||||
name: paperclip-driver # must match the user in the kubeconfig
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: paperclip-tenant-manager
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
```bash
|
||||
# Apply to the workload cluster
|
||||
kubectl apply -f paperclip-driver-external-crb.yaml \
|
||||
--kubeconfig /path/to/workload-cluster-admin.kubeconfig
|
||||
```
|
||||
|
||||
The kubeconfig stored in the Paperclip secret provider must have a `users[].user` entry that matches the `subjects[].name` above. With a certificate-based kubeconfig, the CN in the client certificate is the username.
|
||||
|
||||
When adding the cluster connection:
|
||||
|
||||
```bash
|
||||
paperclipai cluster add \
|
||||
--label workload-cluster-1 \
|
||||
--kind kubeconfig \
|
||||
--kubeconfig-secret vault:secret/data/paperclip/wc1-kubeconfig
|
||||
```
|
||||
|
||||
**Note:** The RBAC rules must be applied to every cluster the kubeconfig user is expected to manage. If a single kubeconfig contains multiple contexts, apply the ClusterRole and ClusterRoleBinding to each cluster independently.
|
||||
|
||||
## Least-privilege notes
|
||||
|
||||
The ClusterRole is broader than M1 strictly requires because it is designed to serve M1 through M2 without an RBAC update. In M1 the driver only provisions namespaces; in M2 it will also schedule Jobs and stream pod logs.
|
||||
|
||||
If your security policy requires strict M1-only RBAC, you can create a narrower ClusterRole that omits `jobs`, `pods`, `pods/log`, `pods/exec`, `persistentvolumeclaims`, and `configmaps`. When M2 ships you will need to re-apply a wider role. This trade-off is yours to make as an operator.
|
||||
|
||||
## Related
|
||||
|
||||
- [Quickstart](./quickstart.md) — first-time cluster setup walkthrough
|
||||
- [Security model](./security-model.md) — how each provisioned object enforces isolation
|
||||
- [Multi-tenant onboarding](./multi-tenant-onboarding.md) — provisioning multiple tenants
|
||||
- [Design spec §2.2](../superpowers/specs/2026-05-08-paperclip-cloud-adapter-design.md#22-pod-identity-zero-trust-by-default)
|
||||
@@ -0,0 +1,257 @@
|
||||
---
|
||||
title: Kubernetes Execution — Multi-Tenant Onboarding
|
||||
summary: Operator playbook for provisioning multiple company namespaces, customising per-tenant quotas, and resolving common edge cases
|
||||
---
|
||||
|
||||
This playbook covers the full lifecycle of onboarding a company onto a Kubernetes cluster: verifying cluster prerequisites, optionally customising the per-tenant policy, running `ensure-tenant`, and verifying the result. The security model behind each provisioned object is described in [security-model.md](./security-model.md).
|
||||
|
||||
**M1 scope:** This playbook covers tenant namespace provisioning only. There is no web UI for these operations yet (M3). Agent execution lands in M2.
|
||||
|
||||
## Step 1 — Verify cluster prerequisites
|
||||
|
||||
Before provisioning any tenant, confirm the cluster is ready.
|
||||
|
||||
```bash
|
||||
paperclipai cluster doctor <clusterId>
|
||||
```
|
||||
|
||||
Check the output for:
|
||||
|
||||
| Check | Expected result |
|
||||
|-------|----------------|
|
||||
| API server reachable | `ok` |
|
||||
| `paperclip-tenant-manager` ClusterRole exists | `ok` |
|
||||
| Default StorageClass present | `ok` — note the StorageClass name for reference |
|
||||
| Cilium detected | `ok` or `not detected` (informational only in M1) |
|
||||
|
||||
If `doctor` reports a missing ClusterRole, apply it:
|
||||
|
||||
```bash
|
||||
kubectl apply -f packages/adapters/kubernetes-execution/manifests/paperclip-tenant-manager-clusterrole.yaml
|
||||
```
|
||||
|
||||
For a from-scratch cluster setup including the ServiceAccount and ClusterRoleBinding, see [quickstart.md](./quickstart.md#first-time-cluster-setup).
|
||||
|
||||
## Step 2 — Optionally customise the per-tenant policy
|
||||
|
||||
By default, every tenant gets the [quota and limit-range defaults](./security-model.md#resourcequota-and-limitrange). If a company needs different resource caps (for example, a larger plan), upsert a row in `cluster_tenant_policies` before running `ensure-tenant`.
|
||||
|
||||
There is no CLI for this in M1. Use a direct database query or a DB migration script.
|
||||
|
||||
### cluster_tenant_policies row shape
|
||||
|
||||
```sql
|
||||
INSERT INTO cluster_tenant_policies (
|
||||
id,
|
||||
cluster_connection_id,
|
||||
company_id,
|
||||
quota_json,
|
||||
limit_range_json,
|
||||
network_json,
|
||||
image_overrides_json,
|
||||
created_at,
|
||||
updated_at
|
||||
) VALUES (
|
||||
gen_random_uuid(),
|
||||
'<clusterId>',
|
||||
'<companyId>',
|
||||
-- quota_json: override any of the default quota fields; null = use defaults
|
||||
'{
|
||||
"requestsCpu": "32",
|
||||
"requestsMemory": "128Gi",
|
||||
"limitsCpu": "128",
|
||||
"limitsMemory": "512Gi",
|
||||
"requestsStorage":"500Gi",
|
||||
"countJobs": 200,
|
||||
"countPvcs": 100,
|
||||
"countSecrets": 400,
|
||||
"countConfigMaps": 400
|
||||
}'::jsonb,
|
||||
-- limit_range_json: override container default/request/max; null = use defaults
|
||||
'{
|
||||
"default": { "cpu": "2", "memory": "4Gi" },
|
||||
"defaultRequest": { "cpu": "500m", "memory": "1Gi" },
|
||||
"max": { "cpu": "16", "memory": "64Gi" },
|
||||
"pvcMaxStorage": "50Gi"
|
||||
}'::jsonb,
|
||||
-- network_json: additional FQDN allowlist entries (Cilium only) and optional HTTP proxy
|
||||
'{
|
||||
"additionalAllowFqdns": ["npm.registry.example.com", "*.internal.example.com"],
|
||||
"httpProxyUrl": null
|
||||
}'::jsonb,
|
||||
-- image_overrides_json: swap runtime images per adapter type; null = use cluster defaults
|
||||
null,
|
||||
now(),
|
||||
now()
|
||||
)
|
||||
ON CONFLICT (cluster_connection_id, company_id) DO UPDATE
|
||||
SET quota_json = EXCLUDED.quota_json,
|
||||
limit_range_json = EXCLUDED.limit_range_json,
|
||||
network_json = EXCLUDED.network_json,
|
||||
image_overrides_json = EXCLUDED.image_overrides_json,
|
||||
updated_at = now();
|
||||
```
|
||||
|
||||
All four JSON columns are nullable. A null value means "use the global default for that field". Individual keys within each JSON object are also optional — omitting a key leaves that specific default unchanged.
|
||||
|
||||
A CLI for managing tenant policies is planned for a future milestone. Until then, the SQL above is the canonical operator interface.
|
||||
|
||||
## Step 3 — Provision the tenant namespace
|
||||
|
||||
```bash
|
||||
paperclipai cluster ensure-tenant <clusterId> <companyId>
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```
|
||||
Provisioned namespace paperclip-<companySlug> (cilium=false)
|
||||
```
|
||||
|
||||
Or if Cilium is detected:
|
||||
|
||||
```
|
||||
Provisioned namespace paperclip-<companySlug> (cilium=true)
|
||||
```
|
||||
|
||||
`ensure-tenant` is idempotent. Running it again on an already-provisioned namespace is safe and will patch any drifted objects back to their desired state. The command reads the `cluster_tenant_policies` row (if it exists) before applying quota and LimitRange objects.
|
||||
|
||||
Objects created (in order):
|
||||
|
||||
1. `Namespace` with PSS labels and `paperclip.ai/*` metadata labels
|
||||
2. `ServiceAccount paperclip-agent` (`automountServiceAccountToken: false`)
|
||||
3. `RoleBinding paperclip-driver` (binds `ClusterRole/paperclip-tenant-manager` to the driver SA, scoped to this namespace)
|
||||
4. `ResourceQuota paperclip-tenant-quota`
|
||||
5. `LimitRange paperclip-tenant-limits`
|
||||
6. `NetworkPolicy default-deny-ingress` and `default-deny-egress`
|
||||
7. `NetworkPolicy paperclip-agent-egress`
|
||||
8. `CiliumNetworkPolicy paperclip-agent-egress-l7` (only when `capabilities.cilium = true`)
|
||||
9. `Secret paperclip-image-pull` (only when image registry credentials are configured)
|
||||
|
||||
## Step 4 — Verify the provisioned namespace
|
||||
|
||||
```bash
|
||||
kubectl describe namespace paperclip-<companySlug>
|
||||
```
|
||||
|
||||
Look for:
|
||||
|
||||
```
|
||||
Labels:
|
||||
paperclip.ai/company-id=<uuid>
|
||||
paperclip.ai/company-slug=<slug>
|
||||
paperclip.ai/managed-by=paperclip
|
||||
pod-security.kubernetes.io/enforce=restricted
|
||||
pod-security.kubernetes.io/audit=restricted
|
||||
pod-security.kubernetes.io/warn=restricted
|
||||
```
|
||||
|
||||
Verify all managed objects are present:
|
||||
|
||||
```bash
|
||||
kubectl get sa,resourcequota,limitrange,networkpolicy \
|
||||
-n paperclip-<companySlug> \
|
||||
-l paperclip.ai/managed-by=paperclip
|
||||
```
|
||||
|
||||
If Cilium is present, also check:
|
||||
|
||||
```bash
|
||||
kubectl get ciliumnetworkpolicy -n paperclip-<companySlug>
|
||||
```
|
||||
|
||||
## Provisioning multiple companies
|
||||
|
||||
Loop over company IDs from your database or source of truth:
|
||||
|
||||
```bash
|
||||
CLUSTER_ID="<clusterId>"
|
||||
|
||||
for COMPANY_ID in \
|
||||
"uuid-company-a" \
|
||||
"uuid-company-b" \
|
||||
"uuid-company-c"; do
|
||||
echo "Provisioning $COMPANY_ID..."
|
||||
paperclipai cluster ensure-tenant "$CLUSTER_ID" "$COMPANY_ID"
|
||||
done
|
||||
```
|
||||
|
||||
`ensure-tenant` is safe to run in parallel; each company operates on a separate namespace.
|
||||
|
||||
## Common edge cases
|
||||
|
||||
### Quota exhaustion
|
||||
|
||||
When an agent run (M2) cannot schedule because the tenant has hit a quota limit, the Kubernetes API returns a `403 Forbidden` with `reason: Forbidden` and a message referencing the quota object. In M1, this surfaces at `ensure-tenant` time only if the quota parameters themselves are invalid (e.g. a LimitRange `max` that is smaller than `default`).
|
||||
|
||||
To inspect current quota usage for a namespace:
|
||||
|
||||
```bash
|
||||
kubectl describe resourcequota paperclip-tenant-quota -n paperclip-<companySlug>
|
||||
```
|
||||
|
||||
To increase quotas, upsert the `cluster_tenant_policies` row with larger values and re-run `ensure-tenant`.
|
||||
|
||||
### DNS resolution issues
|
||||
|
||||
If a pod cannot resolve external hostnames, verify the NetworkPolicy allows DNS egress to `kube-system/kube-dns`:
|
||||
|
||||
```bash
|
||||
kubectl get networkpolicy paperclip-agent-egress \
|
||||
-n paperclip-<companySlug> \
|
||||
-o yaml
|
||||
```
|
||||
|
||||
Look for the DNS egress rule on port 53. If it is missing (e.g. the namespace was provisioned against an older driver), re-run `ensure-tenant` to patch it in.
|
||||
|
||||
Check that CoreDNS is running:
|
||||
|
||||
```bash
|
||||
kubectl get pods -n kube-system -l k8s-app=kube-dns
|
||||
```
|
||||
|
||||
### Image pull failures
|
||||
|
||||
Three distinct failure modes:
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---------|-------------|-----|
|
||||
| `ImagePullBackOff`, event: `no credentials` | No `imagePullSecret` in the namespace | Ensure the cluster connection has an `imageRegistry` configured and re-run `ensure-tenant` |
|
||||
| `ImagePullBackOff`, event: `401 Unauthorized` | Wrong credentials | Rotate the secret in the secret provider and re-run `ensure-tenant` |
|
||||
| `ImagePullBackOff`, event: `i/o timeout` | Registry unreachable | Check the NetworkPolicy allows egress to the registry FQDN on port 443; on Cilium, add the FQDN to `network_json.additionalAllowFqdns` in `cluster_tenant_policies` |
|
||||
|
||||
### Privileged pod admission rejection
|
||||
|
||||
To confirm that PSS Restricted is active and blocking privileged pods:
|
||||
|
||||
```bash
|
||||
kubectl run test-privileged \
|
||||
--image=nginx \
|
||||
--overrides='{"spec":{"containers":[{"name":"test-privileged","image":"nginx","securityContext":{"privileged":true}}]}}' \
|
||||
-n paperclip-<companySlug>
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```
|
||||
Error from server (Forbidden): pods "test-privileged" is forbidden:
|
||||
violates PodSecurity "restricted:latest": ...
|
||||
```
|
||||
|
||||
This rejection proves the PodSecurityAdmission webhook is active on the namespace. If the pod is admitted instead, confirm the PSS labels are present on the namespace with `kubectl describe namespace`.
|
||||
|
||||
### Namespace name collision
|
||||
|
||||
If two companies have the same `companySlug`, the driver uses the fallback name `paperclip-{slug}-{base36hash}`. The `paperclip.ai/company-id` label is always the canonical identifier. To see which namespace belongs to a given company:
|
||||
|
||||
```bash
|
||||
kubectl get ns -l paperclip.ai/company-id=<companyId>
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- [Quickstart](./quickstart.md) — initial cluster setup and first `ensure-tenant`
|
||||
- [Security model](./security-model.md) — what each provisioned object does and why
|
||||
- [Cluster RBAC](./cluster-rbac.md) — driver ClusterRole reference
|
||||
- [Design spec §2.1](../superpowers/specs/2026-05-08-paperclip-cloud-adapter-design.md#21-tenant-boundary-namespace-per-company)
|
||||
- [M1 plan](../superpowers/plans/2026-05-08-paperclip-cloud-adapter-m1-plan.md)
|
||||
@@ -0,0 +1,207 @@
|
||||
---
|
||||
title: Kubernetes Execution — Quickstart
|
||||
summary: First-time setup guide for operators connecting a Kubernetes cluster to Paperclip and provisioning a tenant namespace
|
||||
---
|
||||
|
||||
This guide walks an operator through connecting a Kubernetes cluster to a Paperclip M1 deployment and provisioning the first tenant namespace. By the end you will have a verified cluster connection and a namespace that passes `kubectl describe namespace` — ready for agent execution when M2 ships.
|
||||
|
||||
**M1 scope:** This release delivers tenant namespace provisioning only. Agent execution (running adapter pods) is an M2 feature. See [What's not in M1](#whats-not-in-m1) for the full boundary.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Tool | Version | Notes |
|
||||
|------|---------|-------|
|
||||
| `kubectl` | 1.28+ | Must be on `$PATH` |
|
||||
| `kind` | 0.22+ | Only if using a local kind cluster |
|
||||
| Docker | 24+ | Required to run kind |
|
||||
| Paperclip server | M1 build | `PAPERCLIP_K8S_DRIVER=true` env var must be set |
|
||||
| `paperclipai` CLI | M1 build | `pnpm paperclipai --version` should print a version |
|
||||
|
||||
A PostgreSQL database with the M1 migrations applied (`packages/db/src/migrations/0082_cluster_connections.sql`) is required before any cluster commands work.
|
||||
|
||||
## First-time cluster setup
|
||||
|
||||
### 1. Start a kind cluster (if you don't have one)
|
||||
|
||||
```bash
|
||||
kind create cluster --name paperclip-dev
|
||||
kubectl cluster-info --context kind-paperclip-dev
|
||||
```
|
||||
|
||||
For a production cluster, ensure it has:
|
||||
- A default `StorageClass` (used for agent PVCs in M2)
|
||||
- `NetworkPolicy` enforcement (Calico, Cilium, or compatible CNI)
|
||||
|
||||
### 2. Apply the reference ClusterRole
|
||||
|
||||
The Paperclip driver needs a `ClusterRole` that allows it to manage tenant namespaces and the objects inside them.
|
||||
|
||||
```bash
|
||||
kubectl apply -f packages/adapters/kubernetes-execution/manifests/paperclip-tenant-manager-clusterrole.yaml
|
||||
```
|
||||
|
||||
Verify it was created:
|
||||
|
||||
```bash
|
||||
kubectl get clusterrole paperclip-tenant-manager
|
||||
```
|
||||
|
||||
### 3. Create the driver ServiceAccount and bind the ClusterRole
|
||||
|
||||
For an **in-cluster** topology (Paperclip server running inside the same cluster), create the ServiceAccount in the `paperclip-system` namespace:
|
||||
|
||||
```yaml
|
||||
# paperclip-system-sa.yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: paperclip-system
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: paperclip-driver
|
||||
namespace: paperclip-system
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: paperclip-driver
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: paperclip-driver
|
||||
namespace: paperclip-system
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: paperclip-tenant-manager
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
```bash
|
||||
kubectl apply -f paperclip-system-sa.yaml
|
||||
```
|
||||
|
||||
For a **cross-cluster** topology (Paperclip server is elsewhere), the kubeconfig user must already be bound to the same `ClusterRole` on the workload cluster. See [cluster-rbac.md](./cluster-rbac.md) for details on both topologies.
|
||||
|
||||
### 4. Store the kubeconfig as a Paperclip secret
|
||||
|
||||
Paperclip resolves cluster credentials through its secret provider abstraction. The `kubeconfig_secret_ref` field on a `cluster_connections` row holds a `{ provider, name }` pair. Supported providers in M1:
|
||||
|
||||
| Provider string | Where the secret lives |
|
||||
|-----------------|----------------------|
|
||||
| `env` | Environment variable (e.g. `env:KUBECONFIG_KIND`) |
|
||||
| `aws_secrets` | AWS Secrets Manager secret ARN or name |
|
||||
| `gcp_secret` | GCP Secret Manager resource path |
|
||||
| `vault` | HashiCorp Vault path (`vault:secret/data/paperclip/kind-cfg`) |
|
||||
|
||||
For a local kind cluster, the simplest path is the `env` provider. Export the kubeconfig and set an environment variable on the Paperclip server:
|
||||
|
||||
```bash
|
||||
kind get kubeconfig --name paperclip-dev > /tmp/kind-cfg.yaml
|
||||
export KUBECONFIG_KIND=$(cat /tmp/kind-cfg.yaml)
|
||||
```
|
||||
|
||||
Then reference it as `env:KUBECONFIG_KIND` in the `--kubeconfig-secret` flag below.
|
||||
|
||||
> **Note:** `local_encrypted` (a Paperclip-managed encrypted store) is planned for M2. Until then, use one of the providers listed above.
|
||||
|
||||
## Add a cluster connection
|
||||
|
||||
```bash
|
||||
paperclipai cluster add \
|
||||
--label kind \
|
||||
--kind kubeconfig \
|
||||
--kubeconfig-secret env:KUBECONFIG_KIND
|
||||
```
|
||||
|
||||
The command probes the cluster for capabilities (Cilium presence, default StorageClass, node architectures) and writes a row to `cluster_connections`. The printed `id` is the `<clusterId>` used in subsequent commands.
|
||||
|
||||
Additional flags:
|
||||
|
||||
| Flag | Purpose |
|
||||
|------|---------|
|
||||
| `--paperclip-public-url <url>` | Override the Paperclip server URL agents use to call back (cross-cluster only) |
|
||||
| `--image-registry <url>` | Override the image registry for agent runtime images |
|
||||
|
||||
## Verify the connection
|
||||
|
||||
```bash
|
||||
# Check reachability, RBAC, and cluster capabilities
|
||||
paperclipai cluster doctor <id>
|
||||
|
||||
# List all registered cluster connections
|
||||
paperclipai cluster list
|
||||
```
|
||||
|
||||
`cluster doctor` checks:
|
||||
- API server reachability via the stored kubeconfig
|
||||
- `paperclip-tenant-manager` ClusterRole exists
|
||||
- Default StorageClass is present
|
||||
- Cilium CRD presence (informational)
|
||||
|
||||
## Provision a tenant for a company
|
||||
|
||||
```bash
|
||||
paperclipai cluster ensure-tenant <clusterId> <companyId>
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```
|
||||
Provisioned namespace paperclip-<companySlug> (cilium=false)
|
||||
```
|
||||
|
||||
Or, if Cilium is present on the cluster:
|
||||
|
||||
```
|
||||
Provisioned namespace paperclip-<companySlug> (cilium=true)
|
||||
```
|
||||
|
||||
`ensure-tenant` is **idempotent** — running it twice is safe and brings any drifted objects back to the desired state. The driver refuses to touch a namespace that lacks the `paperclip.ai/managed-by=paperclip` label.
|
||||
|
||||
## Verify the namespace with kubectl
|
||||
|
||||
```bash
|
||||
kubectl get ns,sa,resourcequota,limitrange,networkpolicy \
|
||||
-l paperclip.ai/managed-by=paperclip
|
||||
```
|
||||
|
||||
You should see:
|
||||
- `Namespace` named `paperclip-<companySlug>`
|
||||
- `ServiceAccount` named `paperclip-agent`
|
||||
- `ResourceQuota` named `paperclip-tenant-quota`
|
||||
- `LimitRange` named `paperclip-tenant-limits`
|
||||
- `NetworkPolicy` objects `default-deny-ingress`, `default-deny-egress`, `paperclip-agent-egress`
|
||||
|
||||
```bash
|
||||
kubectl describe namespace paperclip-<companySlug>
|
||||
```
|
||||
|
||||
Check that the PSS labels are present:
|
||||
|
||||
```
|
||||
Labels: pod-security.kubernetes.io/enforce=restricted
|
||||
pod-security.kubernetes.io/audit=restricted
|
||||
pod-security.kubernetes.io/warn=restricted
|
||||
paperclip.ai/managed-by=paperclip
|
||||
paperclip.ai/company-id=<uuid>
|
||||
```
|
||||
|
||||
## What's not in M1
|
||||
|
||||
| Feature | Milestone |
|
||||
|---------|-----------|
|
||||
| Agent execution (running adapter pods as Kubernetes Jobs) | M2 |
|
||||
| Web UI for cluster management and health | M3 |
|
||||
| Per-company BYO cluster (one cluster connection per company) | V2 |
|
||||
| VolumeSnapshot-based agent workspace cloning | V2 |
|
||||
| `local_encrypted` secret provider | M2 |
|
||||
| `paperclipai cluster purge` (namespace teardown) | M2 |
|
||||
|
||||
For the full V1/V2 scope split see the [design spec](../superpowers/specs/2026-05-08-paperclip-cloud-adapter-design.md#non-goals-v1).
|
||||
|
||||
## Next steps
|
||||
|
||||
- [Security model](./security-model.md) — understand the isolation primitives applied to each namespace
|
||||
- [Multi-tenant onboarding](./multi-tenant-onboarding.md) — operator playbook for provisioning multiple companies
|
||||
- [Cluster RBAC](./cluster-rbac.md) — full ClusterRole reference and binding templates
|
||||
@@ -0,0 +1,246 @@
|
||||
---
|
||||
title: Kubernetes Execution — Security Model
|
||||
summary: Isolation primitives applied to every tenant namespace — NetworkPolicy, PodSecurity, RBAC, ResourceQuota, and compliance posture
|
||||
---
|
||||
|
||||
Every company that runs agents on a Kubernetes cluster gets an isolated namespace with a layered set of controls. This document describes each layer, why it exists, and how to verify it. The spec section that defines these primitives is [§2 of the design spec](../superpowers/specs/2026-05-08-paperclip-cloud-adapter-design.md#2-tenancy-isolation--cluster-connection).
|
||||
|
||||
**M1 scope:** These controls are provisioned by `cluster ensure-tenant`. Agent pods that enforce them (PodSecurity Restricted, NetworkPolicy enforcement during actual runs) are an M2 deliverable. The namespace is fully hardened at provision time; the hardening is exercised at run time.
|
||||
|
||||
## Tenancy boundary: one Namespace per company
|
||||
|
||||
Every Paperclip company maps to exactly one namespace per cluster connection, named `paperclip-{companySlug}`. This is the primary isolation boundary: Kubernetes RBAC, ResourceQuota, NetworkPolicy, and PodSecurityAdmission all attach at the namespace level.
|
||||
|
||||
**Naming rules** (from [spec §2.1](../superpowers/specs/2026-05-08-paperclip-cloud-adapter-design.md#21-tenant-boundary-namespace-per-company)):
|
||||
- Primary: `paperclip-{companySlug}` (companySlug truncated to 53 chars so the total stays ≤ 63)
|
||||
- Fallback: `paperclip-{companySlug}-{base36(blake3(companyId))[:8]}` on DNS-1123 overflow or slug collision
|
||||
|
||||
The immutable machine identifier is the `paperclip.ai/company-id=<uuid>` label on the Namespace object, not the namespace name.
|
||||
|
||||
The driver refuses to manage any namespace that does not carry `paperclip.ai/managed-by=paperclip`. This prevents accidental mutation of pre-existing namespaces.
|
||||
|
||||
## Pod identity: zero-trust by default
|
||||
|
||||
Each tenant namespace contains a `ServiceAccount` named `paperclip-agent` with:
|
||||
|
||||
```yaml
|
||||
automountServiceAccountToken: false
|
||||
```
|
||||
|
||||
This ServiceAccount has **no RBAC bindings** by default. A pod running as `paperclip-agent` cannot call the Kubernetes API at all. The driver's own identity (the `paperclip-driver` ServiceAccount in `paperclip-system`) is separate and holds the `ClusterRole` described in [cluster-rbac.md](./cluster-rbac.md).
|
||||
|
||||
Disabling `automountServiceAccountToken` removes the projected token from the pod filesystem, eliminating a common privilege escalation path even if the service account later gains bindings.
|
||||
|
||||
## NetworkPolicy: default-deny + allowlist
|
||||
|
||||
Three NetworkPolicy objects are applied to every tenant namespace.
|
||||
|
||||
### default-deny-ingress
|
||||
|
||||
```yaml
|
||||
podSelector: {}
|
||||
policyTypes: [Ingress]
|
||||
```
|
||||
|
||||
Blocks all ingress to every pod in the namespace. Agent pods do not accept inbound connections.
|
||||
|
||||
### default-deny-egress
|
||||
|
||||
```yaml
|
||||
podSelector: {}
|
||||
policyTypes: [Egress]
|
||||
```
|
||||
|
||||
Blocks all egress by default. Only pods that match the `paperclip-agent-egress` allowlist policy can make outbound connections.
|
||||
|
||||
### paperclip-agent-egress
|
||||
|
||||
Applied only to pods labeled `paperclip.ai/role: agent-runtime`. Three egress rules, in order:
|
||||
|
||||
**1. Cluster DNS**
|
||||
|
||||
```yaml
|
||||
to:
|
||||
- namespaceSelector:
|
||||
matchLabels: { kubernetes.io/metadata.name: kube-system }
|
||||
podSelector:
|
||||
matchLabels: { k8s-app: kube-dns }
|
||||
ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]
|
||||
```
|
||||
|
||||
Agents need name resolution to reach the Paperclip server and external APIs.
|
||||
|
||||
**2. In-cluster Paperclip server** (in-cluster topology only)
|
||||
|
||||
```yaml
|
||||
to:
|
||||
- namespaceSelector:
|
||||
matchLabels: { paperclip.ai/role: control-plane }
|
||||
podSelector:
|
||||
matchLabels: { app.kubernetes.io/name: paperclip-server }
|
||||
ports: [{ port: 443, protocol: TCP }, { port: 3102, protocol: TCP }]
|
||||
```
|
||||
|
||||
Agents call back to the Paperclip server to exchange the bootstrap token for a run JWT. This rule is omitted for cross-cluster topologies where the server is not in the same cluster.
|
||||
|
||||
**3. Internet egress with internal ranges denied**
|
||||
|
||||
```yaml
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
- 169.254.0.0/16 # link-local — cloud metadata service
|
||||
- 100.64.0.0/10 # CGNAT
|
||||
- ipBlock:
|
||||
cidr: ::/0
|
||||
except:
|
||||
- fd00::/8 # IPv6 ULA
|
||||
ports: [{ port: 443, protocol: TCP }]
|
||||
```
|
||||
|
||||
**Why each block matters:**
|
||||
|
||||
| CIDR | What it blocks | Why it matters |
|
||||
|------|----------------|----------------|
|
||||
| `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` | RFC 1918 private ranges | Cluster-internal databases, caches, other services |
|
||||
| `169.254.0.0/16` | Link-local (incl. `169.254.169.254`) | **Cloud metadata service** — this is the primary SSRF target on AWS/GCP/Azure. A compromised pod that can reach the metadata service can obtain instance credentials and escape the tenant boundary |
|
||||
| `100.64.0.0/10` | CGNAT / shared address space | Often used by cloud providers for internal routing; blocks a secondary metadata path on some platforms |
|
||||
| `fd00::/8` | IPv6 ULA | Internal IPv6 ranges; same threat model as RFC 1918. Must be in a separate `ipBlock` entry because the Kubernetes API rejects IPv6 ranges inside an IPv4 cidr |
|
||||
|
||||
The IPv4 and IPv6 deny blocks are intentionally separate `ipBlock` entries. The Kubernetes NetworkPolicy API requires each `except` entry to be a strict subset of its parent `cidr`; mixing IPv6 ranges inside `0.0.0.0/0` is rejected with a 422 error.
|
||||
|
||||
## Cilium variant: FQDN allowlist (auto-detected)
|
||||
|
||||
When `probeClusterCapabilities` detects the `cilium.io/v2` API group, the driver also applies a `CiliumNetworkPolicy` alongside the vanilla NetworkPolicy. The vanilla policy stays as defense-in-depth.
|
||||
|
||||
```yaml
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: paperclip-agent-egress-l7
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels: { paperclip.ai/role: agent-runtime }
|
||||
egress:
|
||||
- toFQDNs:
|
||||
- matchPattern: "*.anthropic.com"
|
||||
- matchPattern: "api.openai.com"
|
||||
- matchPattern: "*.googleapis.com"
|
||||
- matchPattern: "github.com"
|
||||
- matchPattern: "*.github.com"
|
||||
- matchPattern: "gitlab.com"
|
||||
toPorts:
|
||||
- ports: [{ port: "443", protocol: TCP }]
|
||||
- toEndpoints:
|
||||
- matchLabels: { paperclip.ai/role: control-plane }
|
||||
toPorts:
|
||||
- ports: [{ port: "443", protocol: TCP }]
|
||||
```
|
||||
|
||||
The FQDN list is composed from two sources:
|
||||
- `adapter.networkRequirements.allowFqdns` (declared per adapter type in `ServerAdapterModule`)
|
||||
- `cluster_tenant_policies.network_json.additionalAllowFqdns` (per-tenant override, described in [multi-tenant-onboarding.md](./multi-tenant-onboarding.md))
|
||||
|
||||
Cilium detection happens automatically at `cluster add` time. Re-run `cluster doctor <id>` after installing Cilium to update the stored capabilities; then re-run `cluster ensure-tenant` to apply the CNP.
|
||||
|
||||
## PodSecurity Restricted
|
||||
|
||||
The namespace is labeled at provision time:
|
||||
|
||||
```yaml
|
||||
pod-security.kubernetes.io/enforce: restricted
|
||||
pod-security.kubernetes.io/audit: restricted
|
||||
pod-security.kubernetes.io/warn: restricted
|
||||
```
|
||||
|
||||
Any pod admitted to the namespace must conform to the `restricted` PodSecurity Standard. Agent pods (M2) are built to satisfy it:
|
||||
|
||||
```yaml
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
seccompProfile: { type: RuntimeDefault }
|
||||
containers:
|
||||
- securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities: { drop: [ALL] }
|
||||
```
|
||||
|
||||
`readOnlyRootFilesystem: true` is enforced. Writable paths are:
|
||||
- `/workspace` — agent `PersistentVolumeClaim` (M2)
|
||||
- `/tmp` — `emptyDir` volume capped at 1 Gi (M2)
|
||||
|
||||
An attempt to `kubectl run` a privileged pod into a Paperclip namespace will be rejected by the admission controller — this is the expected behavior and confirms PSS is working. See [multi-tenant-onboarding.md](./multi-tenant-onboarding.md#common-edge-cases) for how to verify this.
|
||||
|
||||
## ResourceQuota and LimitRange
|
||||
|
||||
Default values (can be overridden per tenant via `cluster_tenant_policies` — see [multi-tenant-onboarding.md](./multi-tenant-onboarding.md)):
|
||||
|
||||
### ResourceQuota `paperclip-tenant-quota`
|
||||
|
||||
```yaml
|
||||
hard:
|
||||
requests.cpu: "16"
|
||||
requests.memory: "64Gi"
|
||||
limits.cpu: "64"
|
||||
limits.memory: "256Gi"
|
||||
requests.storage: "200Gi"
|
||||
count/jobs.batch: "100"
|
||||
count/persistentvolumeclaims: "50"
|
||||
count/secrets: "200"
|
||||
count/configmaps: "200"
|
||||
```
|
||||
|
||||
### LimitRange `paperclip-tenant-limits`
|
||||
|
||||
```yaml
|
||||
limits:
|
||||
- type: Container
|
||||
default: { cpu: "1", memory: "2Gi" }
|
||||
defaultRequest: { cpu: "250m", memory: "512Mi" }
|
||||
max: { cpu: "8", memory: "32Gi" }
|
||||
- type: PersistentVolumeClaim
|
||||
max: { storage: "20Gi" }
|
||||
```
|
||||
|
||||
The LimitRange ensures that pods without explicit resource requests/limits still get sensible defaults and cannot consume unbounded resources. Without it, a misconfigured pod could exhaust node capacity and starve other tenants.
|
||||
|
||||
## Image pull secret model
|
||||
|
||||
If the cluster connection has an associated image registry and the `imagePullDockerConfigJson` is resolved at provision time, the driver creates a `Secret` of type `kubernetes.io/dockerconfigjson` in the tenant namespace:
|
||||
|
||||
```
|
||||
Secret name: paperclip-image-pull
|
||||
Secret type: kubernetes.io/dockerconfigjson
|
||||
```
|
||||
|
||||
This secret is referenced by agent pod specs (M2) as an `imagePullSecret`. Credentials are per-namespace and are not shared across tenants. The secret value is resolved from the Paperclip secret provider at `ensure-tenant` time.
|
||||
|
||||
## Secret-resolver M1 gap
|
||||
|
||||
The secret provider abstraction supports `aws_secrets`, `gcp_secret`, `vault`, and `env` in M1. The `local_encrypted` provider (Paperclip-managed encrypted store) is planned for M2.
|
||||
|
||||
Operators running a self-hosted Paperclip instance without access to a cloud secret manager should use `env:<VAR_NAME>` for the kubeconfig secret during M1.
|
||||
|
||||
## Compliance bookkeeping
|
||||
|
||||
The M1 provisioning pipeline is aligned with:
|
||||
|
||||
- **NSA/CISA Kubernetes Hardening Guidance** — Restricted PSS, NetworkPolicy default-deny, no privilege escalation, no host network/PID/IPC, drop ALL capabilities, RuntimeDefault seccomp
|
||||
- **CIS Kubernetes Benchmark** — namespace isolation, ResourceQuota enforcement, no automounted ServiceAccount tokens
|
||||
|
||||
The CI release pipeline runs `kube-audit-kit` and `polaris` against a freshly provisioned tenant namespace on every build. PSS Restricted violations or NSA Hardening regressions block the release.
|
||||
|
||||
## Related
|
||||
|
||||
- [Quickstart](./quickstart.md) — set up a cluster connection and run `ensure-tenant`
|
||||
- [Cluster RBAC](./cluster-rbac.md) — the driver ClusterRole and binding templates
|
||||
- [Multi-tenant onboarding](./multi-tenant-onboarding.md) — playbook for provisioning multiple companies and handling edge cases
|
||||
- [Design spec §2](../superpowers/specs/2026-05-08-paperclip-cloud-adapter-design.md#2-tenancy-isolation--cluster-connection)
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,947 @@
|
||||
# Paperclip Cloud Adapter — Multi-Tenant Kubernetes Execution
|
||||
|
||||
**Date:** 2026-05-08
|
||||
**Status:** Spec — pending implementation plan
|
||||
**Owner:** Jannes Stubbemann (brainstorming session)
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Add Kubernetes as a first-class execution target for Paperclip agent runs, with multi-tenant isolation as a design property of the orchestrator (not bolted on later). Every existing adapter (`claude_local`, `codex_local`, `gemini_local`, `opencode_local`, `acpx_local`, `pi_local`, `hermes_local`) gains the ability to run inside a Kubernetes pod by selecting a `kubernetes` execution target — no per-adapter rewrites. Tenancy is enforced via namespace-per-company with the standard k8s-native isolation primitives (RBAC, ResourceQuota, NetworkPolicy, PodSecurity Restricted, per-namespace image pull credentials, ephemeral per-Job Secrets). Each agent gets a persistent `PersistentVolumeClaim` for warm workspaces; each run is an ephemeral `Job`. The orchestrator runs as library code inside the Paperclip server using `@kubernetes/client-node` — no separate operator binary, no CRDs, in V1.
|
||||
|
||||
Cluster topology is hybrid by design: the same code path serves an in-cluster Paperclip (workloads in adjacent namespaces) and a cross-cluster Paperclip (control plane elsewhere; workload cluster reached via stored kubeconfig). Cross-cluster auth reuses the bootstrap-token → run-JWT exchange pattern already specified for the Cursor Cloud adapter.
|
||||
|
||||
The Helm chart for the Paperclip control plane itself is **out of scope for this spec** — it's tracked separately and gated to ship after the cloud adapter lands.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Run Paperclip agents on Kubernetes with strong tenant isolation suitable for multi-tenant SaaS.
|
||||
2. Keep adapters unchanged — extension via the existing `executionTarget` seam, not new adapter types.
|
||||
3. Single shared orchestration package; no per-adapter k8s code.
|
||||
4. Match k8s-native idioms: `Job`, `PVC`, `Namespace`, `NetworkPolicy`, `ResourceQuota`, `LimitRange`, `PodSecurity` Restricted.
|
||||
5. Reuse the bootstrap-token → run-JWT auth flow already shipped for the Cursor Cloud adapter — one server-side route, two callers.
|
||||
6. Support both same-cluster and cross-cluster topologies behind one configuration model.
|
||||
7. Security baseline aligned with NSA/CISA Kubernetes Hardening and CIS Kubernetes Benchmark.
|
||||
8. Operator-debuggable via standard tools: `kubectl get jobs -n paperclip-acme-corp`, audit log entries, structured run-level events.
|
||||
|
||||
## Non-Goals (V1)
|
||||
|
||||
- Helm chart for the Paperclip control plane (separate spec, gated to follow).
|
||||
- Per-company BYO cluster (one or more cluster connections at the *instance* level only).
|
||||
- Pod-per-agent mode (StatefulSet + KEDA scale-to-zero) — designed-for, not built.
|
||||
- External Secrets Operator integration — abstraction in place, no driver in V1.
|
||||
- VolumeSnapshot-based agent cloning.
|
||||
- `PaperclipAgentRun` CRD + reconciliation operator — not built unless reconciliation needs justify it.
|
||||
- Fine-grained image attestation enforcement (we sign; we don't enforce verify in admission).
|
||||
- IPv6 dual-stack pods.
|
||||
|
||||
---
|
||||
|
||||
## Architectural Decisions
|
||||
|
||||
| # | Decision | Rationale |
|
||||
|---|---|---|
|
||||
| 1 | Add a new `kubernetes` kind to the existing `AdapterExecutionTarget` rather than creating a `kubernetes_pod` adapter. | Every adapter inherits k8s execution for free. Single shared orchestrator. Matches the pattern already used for `ssh` and `sandbox`. |
|
||||
| 2 | Tenant boundary = `Namespace` per Paperclip company. | K8s-native isolation primitive. RBAC, ResourceQuota, NetworkPolicy, PSS labels all attach naturally. Free, mature, widely understood. |
|
||||
| 3 | Workload granularity inside a tenant = pod-per-run as a `Job`, with PVC-per-agent for warm workspaces. | Strict isolation per run; PVC reuse keeps workspaces warm without coupling pod lifetimes. Pod-per-company was rejected (poor isolation, doesn't actually save cluster cost). |
|
||||
| 4 | Hybrid topology with bootstrap-token auth. | One auth path serves SaaS (Paperclip on control-plane cluster, agents on workload cluster) and self-hosted single-cluster. Reuses cursor-cloud's exchange route. |
|
||||
| 5 | Imperative orchestration via `@kubernetes/client-node` from the Paperclip server. No CRDs, no operator binary. | Smallest moving parts. Reconciliation is delegated to k8s primitives (Job `backoffLimit`, `TTLAfterFinished`, `OwnerReferences`, `PodFailurePolicy`). Easy to test against `kind`. |
|
||||
| 6 | Workspace bootstrap = init container running the existing Paperclip workspace strategy. | Reuses existing strategy logic; agents see identical filesystem layout to local. Cold = full clone, warm = fetch+reset. |
|
||||
| 7 | Namespace naming = `paperclip-{companySlug}` (operator-friendly), with `-{shortHash}` fallback on collision or DNS-1123 overflow. | Slug already exists in Paperclip (URL `:companyPrefix`). `kubectl` users get human-readable names. Immutable `paperclip.ai/company-id` label is the canonical machine identifier. |
|
||||
| 8 | The platform-module surface is extended (in spec, not yet existing) with `registerExecutionTargetDriver()` to keep parity with `registerAgentAdapter()` / `registerStorageProvider()`. The k8s driver is the first registered driver. | Keeps the door open for third-party drivers (Nomad, ECS, Modal-style) without core changes later. Honors the user's "leverage the plugin system" intent within the actual extension model. |
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture & Code Layout
|
||||
|
||||
### 1.1 The seam
|
||||
|
||||
The codebase already has `AdapterExecutionTarget` in `packages/adapter-utils/src/execution-target.ts` (today: `local`, `ssh`, `sandbox`). Add one new kind:
|
||||
|
||||
```ts
|
||||
type AdapterExecutionTarget =
|
||||
| LocalExecutionTarget
|
||||
| SshExecutionTarget
|
||||
| SandboxExecutionTarget
|
||||
| KubernetesExecutionTarget; // NEW
|
||||
|
||||
interface KubernetesExecutionTarget {
|
||||
kind: "kubernetes";
|
||||
clusterConnectionId: string;
|
||||
namespaceOverride?: string; // rare; defaults to companySlug-derived name
|
||||
imageOverride?: string; // gated by per-cluster policy
|
||||
resources?: { // overrides LimitRange defaults
|
||||
requests?: { cpu?: string; memory?: string };
|
||||
limits?: { cpu?: string; memory?: string };
|
||||
};
|
||||
storage?: {
|
||||
sizeGi?: number; // defaults to per-tenant policy
|
||||
storageClass?: string; // overrides ClusterConnection default
|
||||
};
|
||||
envOverrides?: Record<string, string>; // resolved as secret_refs at materialization
|
||||
}
|
||||
```
|
||||
|
||||
### 1.2 Code layout
|
||||
|
||||
```
|
||||
packages/
|
||||
adapters/
|
||||
kubernetes-execution/ # NEW package: @paperclipai/execution-target-kubernetes
|
||||
package.json
|
||||
tsconfig.json
|
||||
src/
|
||||
index.ts # createKubernetesExecutionDriver()
|
||||
driver.ts # one Job per run, streams stdout, returns AdapterExecutionResult
|
||||
client.ts # @kubernetes/client-node thin wrapper
|
||||
types.ts
|
||||
orchestrator/
|
||||
job.ts # Job spec builder
|
||||
pvc.ts # PVC spec builder
|
||||
secret.ts # ephemeral per-Job Secret materializer (interface + native impl)
|
||||
namespace.ts # company → namespace ensure-and-tag
|
||||
rbac.ts # ServiceAccount + Role + RoleBinding for the namespace
|
||||
network-policy.ts # default-deny + allowlist (vanilla + Cilium variants)
|
||||
resource-quota.ts # ResourceQuota + LimitRange from tenant policy
|
||||
pod-security.ts # restricted PSS context defaults
|
||||
log-stream.ts # k8s log watch → onLog
|
||||
event-watch.ts # Job/Pod Event watch → run log "[k8s]" prefix
|
||||
bootstrap/
|
||||
init-container.ts # workspace-init container spec
|
||||
callback-token.ts # bootstrap-token issuance (calls server)
|
||||
redaction.ts # secret value redaction layer
|
||||
test/
|
||||
unit/ # spec builders, RBAC, NetworkPolicy, redaction
|
||||
integration/ # kind/k3d cluster, full lifecycle
|
||||
|
||||
adapter-utils/
|
||||
src/
|
||||
execution-target.ts # MODIFIED — add KubernetesExecutionTarget kind
|
||||
|
||||
server/
|
||||
src/
|
||||
adapters/
|
||||
execution-target-registry.ts # NEW — platform-module registry for drivers
|
||||
execution-targets/
|
||||
kubernetes.ts # registers @paperclipai/execution-target-kubernetes
|
||||
routes/
|
||||
agent-callback.ts # /api/agent-auth/exchange + /api/runs/:runId/events
|
||||
agent-callback.test.ts # (extend cursor-cloud's existing route)
|
||||
workspace-git-credentials.ts # NEW — /api/workspace/git-credentials
|
||||
services/
|
||||
cluster-connections.ts # NEW — stored cluster connections (kubeconfig refs)
|
||||
cluster-tenant-policies.ts # NEW — quota/limit/image-override policies per tenant
|
||||
workspace-strategy/ # REFACTORED — extracted shared library used by init container
|
||||
|
||||
ui/
|
||||
src/
|
||||
adapters/
|
||||
execution-target/
|
||||
kubernetes-fields.tsx # NEW — exec-target form for "kubernetes"
|
||||
pages/
|
||||
ClusterConnections.tsx # NEW — operator UI: list, add, health
|
||||
ClusterConnectionDetail.tsx # NEW — per-cluster: namespaces, quotas, runs
|
||||
|
||||
docker/
|
||||
agent-runtime/ # NEW — Paperclip-maintained runtime images
|
||||
Dockerfile.base # distroless or ubuntu-slim + node + git + tini + nonroot
|
||||
Dockerfile.claude # base + claude-code CLI
|
||||
Dockerfile.codex
|
||||
Dockerfile.gemini
|
||||
Dockerfile.opencode
|
||||
Dockerfile.acpx
|
||||
Dockerfile.pi
|
||||
Dockerfile.hermes
|
||||
```
|
||||
|
||||
### 1.3 Data flow for one agent run
|
||||
|
||||
```
|
||||
Paperclip server (control plane)
|
||||
|
||||
Heartbeat fires for agent A in company C
|
||||
│
|
||||
▼
|
||||
Adapter.execute(ctx) — ctx.executionTarget.kind === "kubernetes"
|
||||
│
|
||||
▼
|
||||
KubernetesExecutionDriver.run(ctx):
|
||||
1. resolve cluster connection (from ctx.executionTarget.clusterConnectionId)
|
||||
2. ensure namespace `paperclip-{companySlug}` (idempotent)
|
||||
3. ensure RBAC, ResourceQuota, LimitRange, NetworkPolicies, image pull secret
|
||||
4. ensure PVC `agent-{agentSlug}-workspace` (RWO, default StorageClass)
|
||||
5. mint bootstrap token + run JWT scope
|
||||
6. resolve secret_refs → materialize per-Job Secret (OwnerRef → Job)
|
||||
7. submit Job:
|
||||
initContainer paperclip-workspace-init → /workspace via existing strategy
|
||||
container agent-runtime-{adapterType} → exec adapter CLI in /workspace
|
||||
8. open k8s pods/log watch → forward chunks to ctx.onLog("stdout", chunk)
|
||||
9. open k8s Events watch (Job + Pod) → forward warnings to onLog with [k8s] prefix
|
||||
10. await Job completion or cancellation
|
||||
11. read terminal status + map to AdapterExecutionResult (codes per §7.2)
|
||||
12. TTLAfterFinished cleans up Job + Secret; PVC retained
|
||||
▲
|
||||
│ Pod callbacks via /api/agent-auth/exchange + run JWT
|
||||
│
|
||||
Cluster (in-cluster or remote via stored kubeconfig)
|
||||
Namespace: paperclip-{companySlug}
|
||||
Job: agent-{agentSlug}-run-{ulid}
|
||||
Pod
|
||||
initContainer: paperclip-workspace-init (resolves project_workspaces strategy)
|
||||
container: agent-runtime (runs adapter CLI; logs via API)
|
||||
volumes:
|
||||
workspace ← PVC agent-{agentSlug}-workspace (retained between runs)
|
||||
tmp ← emptyDir 1Gi
|
||||
env ← Secret agent-{agentSlug}-run-{ulid}-env (OwnerRef: Job)
|
||||
NetworkPolicy: deny-all + paperclip-agent-egress
|
||||
ResourceQuota / LimitRange (per-tenant policy)
|
||||
```
|
||||
|
||||
### 1.4 Why this shape
|
||||
|
||||
- **Reuses the existing executionTarget seam** — adapters do not change.
|
||||
- **Single trust boundary** — the namespace.
|
||||
- **No long-running daemon** — orchestrator is library code; no operator, no CRDs in V1.
|
||||
- **Built on k8s primitives** — `Job`, `OwnerReference`, `ResourceQuota`, `NetworkPolicy`, `PodSecurityAdmission`. Less code we own.
|
||||
- **Cross-cluster-ready from day one** — cluster connection is a stored kubeconfig ref; in-cluster is one connection type among many.
|
||||
|
||||
---
|
||||
|
||||
## 2. Tenancy, Isolation & Cluster Connection
|
||||
|
||||
### 2.1 Tenant boundary: `Namespace` per company
|
||||
|
||||
**Naming.** `paperclip-{companySlug}` primary. Fallback `paperclip-{companySlug}-{base36(blake3(companyId))[:8]}` when:
|
||||
- slug overflows 63 chars after the `paperclip-` prefix (so slug ≤ 53 chars), or
|
||||
- a different company already owns that name in the same cluster (collision).
|
||||
|
||||
The `paperclip.ai/company-id=<uuid>` label is the immutable machine identifier; the namespace name can change in edge cases without breaking identity.
|
||||
|
||||
**Provisioning.** First time an agent in company C runs against cluster K, the driver runs an idempotent ensure-namespace path that creates (or upserts):
|
||||
|
||||
| Object | Purpose |
|
||||
|---|---|
|
||||
| `Namespace` with labels `paperclip.ai/company-id`, `paperclip.ai/managed-by=paperclip`, PSS labels (`pod-security.kubernetes.io/enforce: restricted`, `audit: restricted`, `warn: restricted`) | Tenant root + admission profile |
|
||||
| `ServiceAccount paperclip-agent` with `automountServiceAccountToken: false` | Pod identity (zero RBAC by default) |
|
||||
| `ResourceQuota paperclip-tenant-quota` | Compute/storage/pod caps |
|
||||
| `LimitRange paperclip-tenant-limits` | Default + max per-pod requests/limits |
|
||||
| `NetworkPolicy default-deny-ingress` + `default-deny-egress` | Zero-trust baseline |
|
||||
| `NetworkPolicy paperclip-agent-egress` (vanilla) | L3/L4 allowlist |
|
||||
| `CiliumNetworkPolicy paperclip-agent-egress-l7` (when Cilium present) | L7/FQDN allowlist |
|
||||
| Optional `Secret paperclip-image-pull` | Per-tenant registry credentials |
|
||||
|
||||
All objects carry `paperclip.ai/managed-by=paperclip` and `paperclip.ai/company-id=<id>`. The driver refuses to mutate any namespace lacking `paperclip.ai/managed-by=paperclip`.
|
||||
|
||||
**Lifecycle.** Company archive → namespace labeled `paperclip.ai/archived=true`, quotas zeroed, retained for the standard data-retention grace period (default 30 days, matches plugin spec § 25.1). Operator purge via `paperclipai cluster purge --company <id>`.
|
||||
|
||||
### 2.2 Pod identity: zero-trust by default
|
||||
|
||||
The pod's ServiceAccount has **no RBAC**. `automountServiceAccountToken: false` is set on the pod spec. The driver's k8s identity (server-side) is a separate ServiceAccount (in-cluster) or kubeconfig user (cross-cluster) bound only to namespaces matching `paperclip-*`.
|
||||
|
||||
### 2.3 NetworkPolicy: default-deny + allowlist (L3/L4)
|
||||
|
||||
```yaml
|
||||
# default-deny-ingress
|
||||
podSelector: {}
|
||||
policyTypes: [Ingress]
|
||||
|
||||
---
|
||||
# default-deny-egress
|
||||
podSelector: {}
|
||||
policyTypes: [Egress]
|
||||
|
||||
---
|
||||
# paperclip-agent-egress (only the agent role)
|
||||
podSelector: { matchLabels: { paperclip.ai/role: agent-runtime } }
|
||||
policyTypes: [Egress]
|
||||
egress:
|
||||
- to: # cluster DNS
|
||||
- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
|
||||
podSelector: { matchLabels: { k8s-app: kube-dns } }
|
||||
ports: [{ port: 53, protocol: UDP }, { port: 53, protocol: TCP }]
|
||||
- to: # in-cluster Paperclip control plane (when topology = same-cluster)
|
||||
- namespaceSelector: { matchLabels: { paperclip.ai/role: control-plane } }
|
||||
podSelector: { matchLabels: { app.kubernetes.io/name: paperclip-server } }
|
||||
ports: [{ port: 443, protocol: TCP }, { port: 3102, protocol: TCP }]
|
||||
- to: # internet egress with internal ranges denied
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
- 169.254.0.0/16 # link-local incl. cloud metadata
|
||||
- 100.64.0.0/10 # CGNAT
|
||||
- fd00::/8 # IPv6 ULA
|
||||
ports: [{ port: 443, protocol: TCP }]
|
||||
```
|
||||
|
||||
The `except` blocks are the load-bearing security control: a compromised pod cannot reach cloud metadata, in-cluster databases, or internal services.
|
||||
|
||||
### 2.4 Cilium variant (auto-detected, additive)
|
||||
|
||||
When the cluster has `CiliumNetworkPolicy` available, the orchestrator generates a CNP **alongside** the vanilla NetworkPolicy. Vanilla stays as defense-in-depth.
|
||||
|
||||
```yaml
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: paperclip-agent-egress-l7
|
||||
spec:
|
||||
endpointSelector: { matchLabels: { paperclip.ai/role: agent-runtime } }
|
||||
egress:
|
||||
- toFQDNs:
|
||||
- matchPattern: "*.anthropic.com"
|
||||
- matchPattern: "api.openai.com"
|
||||
- matchPattern: "*.googleapis.com"
|
||||
- matchPattern: "github.com"
|
||||
- matchPattern: "*.github.com"
|
||||
- matchPattern: "gitlab.com"
|
||||
# ...composed from adapter.networkRequirements + tenantPolicy.additionalAllowFqdns
|
||||
toPorts: [{ ports: [{ port: "443", protocol: TCP }] }]
|
||||
- toEndpoints: [{ matchLabels: { paperclip.ai/role: control-plane } }]
|
||||
toPorts: [{ ports: [{ port: "443", protocol: TCP }] }]
|
||||
```
|
||||
|
||||
### 2.5 Pod Security Admission: `restricted`
|
||||
|
||||
Namespace labeled `pod-security.kubernetes.io/enforce|audit|warn: restricted`. Every pod spec sets:
|
||||
|
||||
```yaml
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
seccompProfile: { type: RuntimeDefault }
|
||||
containers:
|
||||
- securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true # tmpfs at /tmp; PVC at /workspace
|
||||
capabilities: { drop: [ALL] }
|
||||
```
|
||||
|
||||
Satisfies NSA/CISA Kubernetes Hardening Guidance and CIS Kubernetes Benchmark.
|
||||
|
||||
### 2.6 ResourceQuota & LimitRange
|
||||
|
||||
Defaults (overridable per company plan in `cluster_tenant_policies`):
|
||||
|
||||
```yaml
|
||||
# ResourceQuota
|
||||
hard:
|
||||
requests.cpu: "16"
|
||||
requests.memory: "64Gi"
|
||||
limits.cpu: "64"
|
||||
limits.memory: "256Gi"
|
||||
requests.storage: "200Gi"
|
||||
count/jobs.batch: "100"
|
||||
count/persistentvolumeclaims: "50"
|
||||
count/secrets: "200"
|
||||
count/configmaps: "200"
|
||||
|
||||
# LimitRange
|
||||
limits:
|
||||
- type: Container
|
||||
default: { cpu: "1", memory: "2Gi" }
|
||||
defaultRequest: { cpu: "250m", memory: "512Mi" }
|
||||
max: { cpu: "8", memory: "32Gi" }
|
||||
- type: PersistentVolumeClaim
|
||||
max: { storage: "20Gi" }
|
||||
```
|
||||
|
||||
When a billing tier change happens, the driver re-applies the new quota.
|
||||
|
||||
### 2.7 Cluster connection storage
|
||||
|
||||
```ts
|
||||
interface ClusterConnection {
|
||||
id: string;
|
||||
label: string;
|
||||
kind: "in-cluster" | "kubeconfig";
|
||||
kubeconfigSecretRef?: SecretRef; // resolved via Paperclip secret provider
|
||||
apiServerUrl?: string; // for display only
|
||||
defaultNamespacePrefix: string; // default "paperclip-"
|
||||
capabilities: {
|
||||
cilium: boolean; // auto-detected at connect time
|
||||
storageClass: string; // e.g. "gp3", "longhorn"
|
||||
architectures: ("amd64" | "arm64")[];
|
||||
};
|
||||
paperclipPublicUrl?: string; // override for cross-cluster topology
|
||||
imageRegistry?: string; // override for ghcr.io/paperclipai/*
|
||||
createdAt: string;
|
||||
createdBy: string;
|
||||
}
|
||||
```
|
||||
|
||||
V1: instance-level operator-managed cluster connections (`pnpm paperclipai cluster add`). V2: per-company BYO cluster.
|
||||
|
||||
### 2.8 Compliance bookkeeping
|
||||
|
||||
- ✅ NSA/CISA Kubernetes Hardening Guidance — Restricted PSS, NetworkPolicy default-deny, no privilege escalation, no host network/PID/IPC, drop ALL caps, RuntimeDefault seccomp.
|
||||
- ✅ CIS Kubernetes Benchmark — namespace isolation, ResourceQuota, no auto-mounted SA tokens.
|
||||
- ✅ Internal blast-radius isolation — RFC1918 + link-local egress blocked, no shared SA across tenants.
|
||||
|
||||
The release pipeline runs `kube-audit-kit` and `polaris` against a freshly provisioned tenant namespace; PSS Restricted violations or NSA Hardening regressions block release.
|
||||
|
||||
---
|
||||
|
||||
## 3. Pod Lifecycle
|
||||
|
||||
### 3.1 Job spec
|
||||
|
||||
```yaml
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: agent-{agentSlug}-run-{ulid}
|
||||
namespace: paperclip-{companySlug}
|
||||
labels:
|
||||
paperclip.ai/managed-by: paperclip
|
||||
paperclip.ai/company-id: <uuid>
|
||||
paperclip.ai/agent-id: <uuid>
|
||||
paperclip.ai/run-id: <uuid>
|
||||
paperclip.ai/role: agent-runtime
|
||||
spec:
|
||||
backoffLimit: 0 # Paperclip owns retry semantics
|
||||
ttlSecondsAfterFinished: 300 # log harvest, then GC
|
||||
activeDeadlineSeconds: <min(adapterConfig.timeoutSec, namespaceQuota.maxRunSeconds)>
|
||||
completions: 1
|
||||
parallelism: 1
|
||||
podFailurePolicy:
|
||||
rules:
|
||||
- action: FailJob
|
||||
onPodConditions: [{ type: PodHasNetwork, status: "False" }]
|
||||
- action: FailJob
|
||||
onExitCodes: { containerName: agent, operator: In, values: [137] } # OOM
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
paperclip.ai/role: agent-runtime
|
||||
paperclip.ai/agent-id: <uuid>
|
||||
paperclip.ai/run-id: <uuid>
|
||||
annotations:
|
||||
paperclip.ai/job-spec-version: "v1"
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
serviceAccountName: paperclip-agent
|
||||
restartPolicy: Never
|
||||
enableServiceLinks: false
|
||||
terminationGracePeriodSeconds: 30
|
||||
securityContext: { ... restricted PSS ... }
|
||||
initContainers: [ workspace-init ]
|
||||
containers: [ agent ]
|
||||
volumes:
|
||||
- name: workspace
|
||||
persistentVolumeClaim: { claimName: agent-{agentSlug}-workspace }
|
||||
- name: tmp
|
||||
emptyDir: { sizeLimit: 1Gi }
|
||||
- name: env
|
||||
secret: { secretName: agent-{agentSlug}-run-{ulid}-env, defaultMode: 0400 }
|
||||
- name: skills-pointer
|
||||
configMap: { name: paperclip-skills-pointer }
|
||||
```
|
||||
|
||||
Key choices:
|
||||
- `backoffLimit: 0` — Paperclip owns retry semantics (`AdapterExecutionErrorFamily`, `retryNotBefore`). K8s never retries.
|
||||
- `ttlSecondsAfterFinished: 300` — gives log-watch time to drain final stdout, then GC removes Job + Pod + per-run Secret via `OwnerReferences`.
|
||||
- `activeDeadlineSeconds` — clamped to `min(adapterConfig.timeoutSec, namespaceQuota.maxRunSeconds)`.
|
||||
- `podFailurePolicy` — surfaces image-pull and OOM as terminal failures rather than consuming retry budget.
|
||||
- `automountServiceAccountToken: false` — agent has no business calling the k8s API.
|
||||
- `enableServiceLinks: false` — avoids env-var noise and minor info leak.
|
||||
|
||||
### 3.2 Init container: workspace bootstrap
|
||||
|
||||
```yaml
|
||||
initContainers:
|
||||
- name: workspace-init
|
||||
image: ghcr.io/paperclipai/agent-runtime-base:{paperclipVersion}
|
||||
command: ["/usr/local/bin/paperclip-workspace-init"]
|
||||
env:
|
||||
- name: PAPERCLIP_WORKSPACE_STRATEGY
|
||||
value: <serialized strategy from project_workspaces>
|
||||
- name: PAPERCLIP_WORKSPACE_ROOT
|
||||
value: /workspace
|
||||
- name: PAPERCLIP_RUN_ID
|
||||
value: <run-id>
|
||||
- name: PAPERCLIP_BOOTSTRAP_TOKEN
|
||||
valueFrom: { secretKeyRef: { name: agent-{agentSlug}-run-{ulid}-env, key: BOOTSTRAP_TOKEN } }
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
- { name: tmp, mountPath: /tmp }
|
||||
securityContext: { ... restricted ... }
|
||||
resources:
|
||||
requests: { cpu: 200m, memory: 256Mi }
|
||||
limits: { cpu: "2", memory: 1Gi }
|
||||
```
|
||||
|
||||
`paperclip-workspace-init` behavior:
|
||||
|
||||
1. Reads `PAPERCLIP_WORKSPACE_STRATEGY` (existing strategy types: `git-clone`, `git-worktree`, `existing-path` (rejected at validation), `none`).
|
||||
2. Cold PVC → full strategy. Git creds obtained via bootstrap-token → run-JWT exchange → `/api/workspace/git-credentials`. Creds written to init container's tmpfs only.
|
||||
3. Warm PVC → `git fetch && git reset --hard {ref}` (configurable; matches local adapter semantics).
|
||||
4. Writes `.paperclip-workspace-state.json` marker.
|
||||
5. Non-zero exit → Job fails with `errorCode: workspace_init_failed`.
|
||||
|
||||
Init container shares the PVC with the main container but **not** the env Secret mount — credentials never leak forward.
|
||||
|
||||
### 3.3 Main container: agent runtime
|
||||
|
||||
```yaml
|
||||
containers:
|
||||
- name: agent
|
||||
image: <resolved per §5.2>
|
||||
imagePullPolicy: IfNotPresent
|
||||
workingDir: /workspace
|
||||
command: ["/usr/local/bin/tini", "--"]
|
||||
args: ["/usr/local/bin/paperclip-agent-shim", "--adapter", "<type>"]
|
||||
env:
|
||||
- name: PAPERCLIP_RUN_ID
|
||||
value: <run-id>
|
||||
- name: PAPERCLIP_PUBLIC_URL
|
||||
value: <resolved per §6.5>
|
||||
- name: PAPERCLIP_BOOTSTRAP_TOKEN
|
||||
valueFrom: { secretKeyRef: { name: ...env, key: BOOTSTRAP_TOKEN } }
|
||||
- name: TRACEPARENT
|
||||
value: <propagated from server span>
|
||||
# ...adapter-specific keys (LLM keys, etc.) from the per-Job Secret
|
||||
volumeMounts:
|
||||
- { name: workspace, mountPath: /workspace }
|
||||
- { name: tmp, mountPath: /tmp }
|
||||
- { name: env, mountPath: /run/paperclip/env, readOnly: true }
|
||||
- { name: skills-pointer, mountPath: /run/paperclip/skills, readOnly: true }
|
||||
resources:
|
||||
requests: { cpu: <from adapter or LimitRange default>, memory: ... }
|
||||
limits: { cpu: ..., memory: ... }
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
readOnlyRootFilesystem: true
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities: { drop: [ALL] }
|
||||
seccompProfile: { type: RuntimeDefault }
|
||||
```
|
||||
|
||||
`tini` as PID 1 ensures SIGTERM forwarding. `paperclip-agent-shim`:
|
||||
1. Reads `AdapterRuntimeCommandSpec` from a config file projected by the orchestrator.
|
||||
2. Detects the adapter CLI (`installCommand` if missing).
|
||||
3. Reads the prerendered prompt (orchestrator-rendered).
|
||||
4. Exec-replaces itself into the actual adapter CLI so signals propagate.
|
||||
5. Frames structured stdout events compatible with existing UI parsers.
|
||||
|
||||
No sidecars in V1. Logs flow via `pods/log` watch + structured-events callback (§3.4).
|
||||
|
||||
### 3.4 Output streaming
|
||||
|
||||
Two parallel streams merged by timestamp:
|
||||
|
||||
| Source | Mechanism |
|
||||
|---|---|
|
||||
| stdout/stderr | `pods/log` watch with `sinceTime` reconnect → `ctx.onLog("stdout", chunk)` |
|
||||
| Structured events | `POST /api/runs/:runId/events` from agent shim with run JWT |
|
||||
| K8s `Event`s on Job/Pod | Events watch → `ctx.onLog("stdout", "[k8s] " + event)` |
|
||||
|
||||
If the structured-events callback is unreachable (NetworkPolicy misconfig), events fall back to stdout framing — degraded but not broken.
|
||||
|
||||
### 3.5 Cancellation
|
||||
|
||||
User cancels run → registered cancellation handler:
|
||||
1. `kubectl delete job <name> --propagation-policy=Foreground --grace-period=30`
|
||||
2. K8s sends SIGTERM to PID 1 → `tini` → adapter CLI. 30s drain.
|
||||
3. After grace, SIGKILL.
|
||||
4. Pod terminates → log watch sees stream end → `AdapterExecutionResult { exitCode: null, signal: "SIGTERM", errorCode: "cancelled" }`.
|
||||
|
||||
Foreground propagation ensures the Pod is gone before the API call returns; per-Job Secret GC'd via `OwnerReferences`. PVC untouched.
|
||||
|
||||
### 3.6 Concurrent runs on the same agent
|
||||
|
||||
PVC is `ReadWriteOnce` — only one pod can mount at a time.
|
||||
- Paperclip's heartbeat-level locking already prevents overlapping runs per agent.
|
||||
- Defensive: orchestrator checks for a live Job with the same `paperclip.ai/agent-id` label and returns `errorCode: "concurrent_run_blocked"` immediately rather than queueing.
|
||||
- For agents that legitimately need concurrent runs, switch the agent's storage class to ReadWriteMany (e.g. EFS, Azure Files, Longhorn) with the perf tradeoff documented.
|
||||
|
||||
---
|
||||
|
||||
## 4. Workspace Persistence
|
||||
|
||||
### 4.1 PVC lifetime is bound to the agent, not the run
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: agent-{agentSlug}-workspace
|
||||
namespace: paperclip-{companySlug}
|
||||
labels:
|
||||
paperclip.ai/managed-by: paperclip
|
||||
paperclip.ai/company-id: <uuid>
|
||||
paperclip.ai/agent-id: <uuid>
|
||||
paperclip.ai/role: agent-workspace
|
||||
annotations:
|
||||
paperclip.ai/workspace-strategy: <strategy-key>
|
||||
paperclip.ai/created-at: <iso>
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: <from ClusterConnection.capabilities.storageClass>
|
||||
resources: { requests: { storage: <perAgentDefault, e.g. 10Gi> } }
|
||||
```
|
||||
|
||||
Created on first run, reused for subsequent runs. Never auto-deleted on Job completion.
|
||||
|
||||
### 4.2 Zonal pinning
|
||||
|
||||
`ReadWriteOnce` cloud disks (EBS, PD-Standard, Azure Disk) are zone-bound. After first bind, the orchestrator reads the bound `PersistentVolume.spec.nodeAffinity` and adds matching `nodeAffinity` to subsequent pods so they land in the same zone. Operators who want multi-zone resilience choose a regional StorageClass (e.g. `pd-balanced` regional, EFS, Longhorn) — purely a StorageClass decision, no orchestrator code change.
|
||||
|
||||
### 4.3 Reclaim & GC
|
||||
|
||||
- PVC `persistentVolumeReclaimPolicy: Delete` (default for dynamic provisioning).
|
||||
- Agent archive → PVC labeled `paperclip.ai/archived=true`, retained for grace period (default 30 days).
|
||||
- Operator `paperclipai cluster purge --agent <id>` → immediate delete.
|
||||
- Company purge → namespace deletion cascades to all PVCs.
|
||||
- Daily stale PVC sweep flags PVCs with no matching `agents` row older than 7 days; surfaced in cluster-health UI; never auto-deleted.
|
||||
|
||||
### 4.4 Quota interaction
|
||||
|
||||
PVC storage counts against `requests.storage` and `count/persistentvolumeclaims` in the namespace `ResourceQuota`. Quota exhaustion → new agents cannot run; orchestrator returns `errorCode: "tenant_storage_quota_exhausted"` with remediation hint.
|
||||
|
||||
### 4.5 Workspace strategy → init container interaction
|
||||
|
||||
| Strategy | Init container behavior |
|
||||
|---|---|
|
||||
| `git-clone` | Cold: `git clone <url> --branch <ref> .`. Warm: `git fetch && git reset --hard origin/<ref>`. |
|
||||
| `git-worktree` | Cold: `git clone --bare` into `/workspace/.bare` then `git worktree add` per agent slot. Warm: `git fetch` + worktree update. |
|
||||
| `existing-path` | Rejected at execution-target config validation; not a runtime failure. |
|
||||
| `none` | Empty `/workspace`. |
|
||||
|
||||
Strategy code is **shared** between local adapter and init container by extracting existing logic from `server/src/services/workspace-strategy/` (or wherever it currently lives) into a small library that the init container's binary links against. This is a planned refactor in the implementation plan.
|
||||
|
||||
### 4.6 Snapshots & backup (V2, designed-for)
|
||||
|
||||
- PVCs labeled and annotated for Velero/Kasten K10 selectors out of the box.
|
||||
- VolumeSnapshot CRDs reserved for V2 — `paperclipai cluster snapshot agent <id>` would create a `VolumeSnapshot` and a Paperclip record.
|
||||
- V2 use case: cloning an agent's workspace as the seed for a new agent (`PVC.dataSource: VolumeSnapshot`).
|
||||
|
||||
### 4.7 Edge cases
|
||||
|
||||
- Init container wedge → `activeDeadlineSeconds` catches.
|
||||
- Disk full mid-run → `errorCode: workspace_disk_full` with `df` snapshot from `kubectl exec`.
|
||||
- Corrupt workspace → init container detects `git status` non-clean and falls back to full reset; configurable per agent for agents that want dirty state preserved.
|
||||
- PVC orphan after stuck `Terminating` namespace → surfaced in cluster-health UI; explicit operator cleanup, never auto-resolved.
|
||||
|
||||
---
|
||||
|
||||
## 5. Images & Secrets
|
||||
|
||||
### 5.1 Image strategy: Paperclip-maintained, adapter-aligned
|
||||
|
||||
Family of small images, not one fat image:
|
||||
|
||||
| Image | Base | Contains |
|
||||
|---|---|---|
|
||||
| `ghcr.io/paperclipai/agent-runtime-base:{paperclipVersion}` | distroless or ubuntu-slim | `tini`, `git`, `paperclip-workspace-init`, `paperclip-agent-shim`, CA bundle, non-root uid/gid 1000 |
|
||||
| `agent-runtime-claude:{paperclipVersion}` | base | + `@anthropic-ai/claude-code` CLI |
|
||||
| `agent-runtime-codex:{paperclipVersion}` | base | + Codex CLI |
|
||||
| `agent-runtime-gemini:{paperclipVersion}` | base | + Gemini CLI |
|
||||
| `agent-runtime-opencode:{paperclipVersion}` | base | + OpenCode CLI |
|
||||
| `agent-runtime-acpx:{paperclipVersion}` | base | + ACPX CLI |
|
||||
| `agent-runtime-pi:{paperclipVersion}` | base | + Pi CLI |
|
||||
| `agent-runtime-hermes:{paperclipVersion}` | base | + Hermes CLI |
|
||||
|
||||
Reasoning:
|
||||
- Per-adapter images are smaller (~150–250 MB) than a fat kitchen sink image.
|
||||
- Independent CLI version pinning per adapter.
|
||||
- Clear failure surface: "Codex CLI not found in image" is impossible because Codex isn't in the Claude image.
|
||||
|
||||
`{paperclipVersion}` ties image **tags** to Paperclip release tags so `agent-runtime-claude:v2026.5.8` is unambiguous. The image *contents* (notably the bundled adapter CLI version) follow each adapter's own pinning cadence — a Paperclip release that doesn't bump the Claude CLI ships an `agent-runtime-claude:v2026.5.8` whose layered content is identical to the previous release. This keeps tag→content mapping deterministic without forcing a CLI rebuild on every Paperclip release.
|
||||
|
||||
**Multi-arch.** Every image ships `amd64` + `arm64` via `docker buildx`.
|
||||
|
||||
**Provenance.** `cosign` keyless OIDC signing in CI; SBOMs (`syft` → SPDX) attached as cosign attestations; `trivy` CVE scanning gates release.
|
||||
|
||||
### 5.2 Image resolution: three levels of override
|
||||
|
||||
```
|
||||
Effective image =
|
||||
per-agent override (adapterConfig.kubernetes.image; gated by per-cluster policy)
|
||||
?? per-tenant override (cluster_tenant_policies.imageOverrides[adapterType])
|
||||
?? cluster default (ClusterConnection.imageRegistry + adapterType)
|
||||
?? Paperclip default (ghcr.io/paperclipai/agent-runtime-{adapterType}:{paperclipVersion})
|
||||
```
|
||||
|
||||
Per-agent override is gated by `ClusterConnection.allowAgentImageOverride: false` (default false). Even when enabled, the image must satisfy `imagePullPolicy: IfNotPresent` and an admission check that the namespace is not labeled `paperclip.ai/role: control-plane`.
|
||||
|
||||
### 5.3 Per-namespace image pull credentials
|
||||
|
||||
Per-namespace `kubernetes.io/dockerconfigjson` Secret resolved from the Paperclip secret store at namespace-ensure time. Not auto-rotated (rotation can break in-flight pulls); operator command `paperclipai cluster rotate-pull-secret --connection <id>` for explicit rotation.
|
||||
|
||||
### 5.4 Per-Job ephemeral Secret for run credentials
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
type: Opaque
|
||||
metadata:
|
||||
name: agent-{agentSlug}-run-{ulid}-env
|
||||
namespace: paperclip-{companySlug}
|
||||
labels:
|
||||
paperclip.ai/managed-by: paperclip
|
||||
paperclip.ai/run-id: <uuid>
|
||||
ownerReferences:
|
||||
- apiVersion: batch/v1
|
||||
kind: Job
|
||||
name: agent-{agentSlug}-run-{ulid}
|
||||
uid: <jobUid>
|
||||
controller: true
|
||||
blockOwnerDeletion: true
|
||||
data:
|
||||
BOOTSTRAP_TOKEN: <base64>
|
||||
ANTHROPIC_API_KEY: <base64> # resolved from secret_ref
|
||||
GIT_CREDENTIAL_KEY: <base64> # short-lived handle, exchanged at runtime
|
||||
# ...adapter-required keys, all from secret_refs
|
||||
```
|
||||
|
||||
Properties:
|
||||
- Lifetime tied to the Job via `OwnerReferences` — TTL cleanup of the Job removes the Secret.
|
||||
- Mounted as **files** at `/run/paperclip/env` (mode 0400, uid 1000), and projected as env vars only for the keys the adapter CLI explicitly needs.
|
||||
- Materialized at Job-create time (resolve `secret_ref`s, build Secret, submit Job in a single transaction). On Job-create failure, Secret is explicitly deleted (no owner yet to GC it).
|
||||
- Secret values never written to logs, audit entries, error messages, or `resultJson`. Enforced by a redaction layer keyed by the materialized Secret's value set.
|
||||
|
||||
### 5.5 Bootstrap token → run JWT exchange (shared with cursor-cloud)
|
||||
|
||||
Same shape as the existing cursor-cloud route at `/api/agent-auth/exchange`:
|
||||
1. Pod has `BOOTSTRAP_TOKEN` (10 min TTL, run-scoped, single-use, bound to Job UID).
|
||||
2. `POST /api/agent-auth/exchange { bootstrapToken }` → `{ runJwt, expiresAt }`.
|
||||
3. Pod uses `runJwt` for `/api/runs/:runId/events`, `/api/skills/*`, `/api/workspace/git-credentials`.
|
||||
|
||||
V1.5 second factor: bind the exchange request to the calling Pod's projected ServiceAccount token (`audience: paperclip-runtime`); server's `TokenReview` confirms the caller before issuing the run JWT. Cross-cluster topology defers this to V2 (needs identity federation).
|
||||
|
||||
### 5.6 External Secrets Operator (V2, designed-for)
|
||||
|
||||
V1 abstracts the per-Job Secret materialization behind a `SecretMaterializer` interface. V2 adds an `ExternalSecretMaterializer` that creates an `ExternalSecret` CR pointing at the customer's Vault/AWS SM/GCP SM and waits for ESO to materialize the underlying Secret before submitting the Job.
|
||||
|
||||
### 5.7 Not in V1
|
||||
|
||||
- Per-pod kubelet credential provider plugins.
|
||||
- Paperclip-managed cosign verification policy (we sign; we trust customers' admission controllers to verify).
|
||||
- Image rebuild on every Paperclip release for every adapter CLI version (CLI bumps follow their own cadence).
|
||||
|
||||
---
|
||||
|
||||
## 6. Networking & Callback
|
||||
|
||||
### 6.1 Three layers of egress control
|
||||
|
||||
```
|
||||
Pod (agent-runtime)
|
||||
↓ ① NetworkPolicy (L3/L4 IP/port) ← always on
|
||||
↓ ② Cilium L7 / FQDN policy ← if cluster supports it (auto-detected)
|
||||
↓ ③ Egress proxy (squid/Envoy) ← optional, customer-managed
|
||||
↓
|
||||
external network
|
||||
```
|
||||
|
||||
V1 ships ① and ②. ③ is documented; the runtime image honors `HTTP_PROXY`/`HTTPS_PROXY` env so customers can wire one in via tenant policy.
|
||||
|
||||
### 6.2 Layer ① — vanilla NetworkPolicy
|
||||
|
||||
See §2.3 for the full policies. The control-plane match-labels are a parameter of the cluster connection: in-cluster topology fills them with the Paperclip server's labels; cross-cluster omits the in-cluster rule entirely.
|
||||
|
||||
### 6.3 Layer ② — Cilium variant
|
||||
|
||||
See §2.4. The FQDN allowlist is per-tenant + per-adapter:
|
||||
- `adapter.networkRequirements.allowFqdns` (declared by the adapter package)
|
||||
- `tenantPolicy.additionalAllowFqdns` (operator-set per company)
|
||||
- Deny-by-default everywhere else.
|
||||
|
||||
### 6.4 Layer ③ — egress proxy
|
||||
|
||||
Tenant policy can set `httpProxyUrl: http://proxy.acme-corp.svc.cluster.local:3128`. Orchestrator injects `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`; `NO_PROXY` is auto-populated with kube-dns, the Paperclip Service, and pod/service CIDRs.
|
||||
|
||||
### 6.5 Paperclip public URL resolution
|
||||
|
||||
Resolution order:
|
||||
1. **In-cluster topology** → `https://paperclip-server.<paperclipNamespace>.svc.cluster.local:443`. Default when `ClusterConnection.kind === "in-cluster"` and the orchestrator can resolve the server's Service.
|
||||
2. **Cross-cluster topology** → `ClusterConnection.paperclipPublicUrl`, fallback `process.env.PAPERCLIP_PUBLIC_URL`. Validated reachable from the cluster at connection-add time via a one-shot Job that curls `/api/health`.
|
||||
3. **Per-agent override** — `adapterConfig.paperclipPublicUrl`. Rare; for agents crossing VPN boundaries.
|
||||
|
||||
The resolved URL is injected as a literal `PAPERCLIP_PUBLIC_URL` env var in the pod spec (not in the per-Job Secret — it's not sensitive). The agent shim refuses to start if it's missing.
|
||||
|
||||
### 6.6 Callback API surface
|
||||
|
||||
| Endpoint | Auth | Used by | Source |
|
||||
|---|---|---|---|
|
||||
| `POST /api/agent-auth/exchange` | bootstrap token | first call from pod | shared with cursor-cloud |
|
||||
| `POST /api/runs/:runId/events` | run JWT | structured events | shared |
|
||||
| `POST /api/workspace/git-credentials` | run JWT | init container | k8s-specific |
|
||||
|
||||
Rate limits on `/api/agent-auth/exchange`: 10/min/companyId, 1000/day/companyId. Bootstrap tokens are single-use; replay → `400 token_already_consumed`. Run JWT bearer-auth on subsequent calls; per-run rate limits 1000 events/min, 100MB/run total.
|
||||
|
||||
### 6.7 Failure modes
|
||||
|
||||
- DNS down inside cluster → `errorCode: dns_unreachable` with CoreDNS pod status in metadata.
|
||||
- Cilium policy denies a new domain → drop logs (Hubble) tee'd into run log when available; `errorCode: network_policy_denied`.
|
||||
- Paperclip server unreachable → bounded backoff (5 attempts, exponential, capped at 30s); after that `errorCode: control_plane_unreachable`, `errorFamily: transient_upstream`.
|
||||
- Token replay → orchestrator-internal retry once with fresh token, then fail.
|
||||
|
||||
---
|
||||
|
||||
## 7. Observability, Failure Modes, Testing
|
||||
|
||||
### 7.1 Logs
|
||||
|
||||
Three tiers feeding the Paperclip run log buffer:
|
||||
|
||||
| Source | Mechanism | Latency |
|
||||
|---|---|---|
|
||||
| Agent stdout/stderr | `pods/log` watch | live (~100ms) |
|
||||
| Structured events | `POST /api/runs/:runId/events` from agent shim | live |
|
||||
| Orchestrator + K8s `Event`s | server-local logs + `heartbeat_run_events` rows + `[k8s]`-prefixed lines | sync |
|
||||
|
||||
Tier 3 catches things teams forget: a Job sitting Pending for 90s due to `ImagePullBackOff` shows up in the run log without `kubectl describe`.
|
||||
|
||||
### 7.2 Metrics (Prometheus, exposed at server `/metrics`)
|
||||
|
||||
```
|
||||
paperclip_k8s_runs_total{cluster, namespace, adapter_type, outcome}
|
||||
paperclip_k8s_run_duration_seconds{cluster, namespace, adapter_type}
|
||||
paperclip_k8s_pod_pending_seconds{cluster, namespace}
|
||||
paperclip_k8s_pvc_bytes{cluster, namespace, agent_id}
|
||||
paperclip_k8s_namespace_quota_used_ratio{cluster, namespace, resource}
|
||||
paperclip_k8s_orchestrator_api_errors_total{cluster, verb, code}
|
||||
paperclip_k8s_callback_requests_total{endpoint, status_code}
|
||||
```
|
||||
|
||||
Customers worried about cardinality can drop the `namespace` label via Helm values.
|
||||
|
||||
### 7.3 Tracing (OpenTelemetry)
|
||||
|
||||
Server starts a span on `Adapter.execute`, propagates context as `TRACEPARENT` env into the per-Job Secret. Agent shim continues the trace. One trace per run with spans for orchestrator phases (ensure-namespace, materialize-secret, submit-job, await-completion) plus agent-side spans for prompt rendering and CLI invocation.
|
||||
|
||||
### 7.4 Audit
|
||||
|
||||
Every orchestrator mutation writes a Paperclip activity log entry with:
|
||||
- `actorType: "platform_module"`
|
||||
- `sourceModule: "kubernetes-execution-target"`
|
||||
- `targetCluster`, `targetNamespace`, `verb`, `outcome`
|
||||
|
||||
Sufficient for SOC2-style audit trails.
|
||||
|
||||
### 7.5 Failure mode catalog
|
||||
|
||||
| Symptom | `errorCode` | Family | Retryable? |
|
||||
|---|---|---|---|
|
||||
| Image pull fails | `image_pull_failed` | `transient_upstream` | No (operator must fix image/pull-secret) |
|
||||
| OOM kill | `oom_killed` | — | No (resize agent's resource request) |
|
||||
| `activeDeadlineSeconds` hit | `timeout` | — | Per Paperclip's existing timeout policy |
|
||||
| Init container fails | `workspace_init_failed` | varies | Yes if `transient_upstream` (e.g. git server 503) |
|
||||
| Disk full | `workspace_disk_full` | — | No (PVC resize or workspace cleanup) |
|
||||
| Storage quota | `tenant_storage_quota_exhausted` | — | No (billing event) |
|
||||
| Compute quota | `tenant_compute_quota_exhausted` | — | No (billing event) |
|
||||
| Cluster unreachable from server | `cluster_unreachable` | `transient_upstream` | Yes |
|
||||
| Pod can't reach Paperclip | `control_plane_unreachable` | `transient_upstream` | Yes (with backoff) |
|
||||
| Cilium FQDN denied | `network_policy_denied` | — | No (operator must update policy) |
|
||||
| Concurrent run blocked | `concurrent_run_blocked` | — | Yes after current run completes |
|
||||
| Bootstrap token replay | `token_replay` | — | No (orchestrator retries once internally, then fails) |
|
||||
| Pod stuck `Pending` >5min | `pod_scheduling_failed` | `transient_upstream` | Yes (autoscaler may catch up) |
|
||||
| Workspace strategy unsupported on this target | `execution_target_unsupported_strategy` | — | No (config-validation-time error) |
|
||||
| DNS unreachable | `dns_unreachable` | `transient_upstream` | Yes |
|
||||
| User-initiated cancellation | `cancelled` | — | No (terminal by user intent) |
|
||||
|
||||
Every code is documented with an inline UI remediation hint.
|
||||
|
||||
### 7.6 Testing strategy
|
||||
|
||||
**Unit (no cluster).** Bulk of suite. Pure builders:
|
||||
- Job spec builder → expected YAML (golden snapshots).
|
||||
- NetworkPolicy generator (vanilla + Cilium).
|
||||
- ResourceQuota / LimitRange builder per plan tier.
|
||||
- RBAC binding generator.
|
||||
- Secret materialization → no plaintext leakage in any return value.
|
||||
- ClusterConnection validation, namespace-name derivation, slug truncation.
|
||||
- Redaction layer on log lines, error messages, `resultJson`.
|
||||
|
||||
**Integration (real cluster).** `kind`/`k3d` cluster spun up in CI via `testcontainers-node`:
|
||||
- Full run lifecycle for `claude_local` against a fake LLM endpoint.
|
||||
- Cancellation mid-run — Job deleted within grace, no orphan PVC/Secret.
|
||||
- Quota enforcement — submit 11 Jobs with quota of 10; 11th rejected with the right error code.
|
||||
- Multi-tenant isolation — two namespaces, concurrent agents, probe pod that tries cross-namespace traffic must fail.
|
||||
- PSS Restricted compliance — privileged Pod rejected at admission.
|
||||
- Image pull failure path → `errorCode: image_pull_failed` within 60s.
|
||||
- Workspace warm vs cold — second run faster, PVC reused.
|
||||
|
||||
**Contract.** Every public type the driver exports has a contract test; the execution-target driver registry interface is the most important.
|
||||
|
||||
**Security review gate.** `kube-audit-kit` and `polaris` run against a freshly provisioned tenant namespace in CI. PSS Restricted violations or NSA Hardening regressions block release.
|
||||
|
||||
**Load.** Synthetic 100-concurrent-run test against `kind` to validate the orchestrator's k8s API call rate stays under default `--max-requests-inflight`. The orchestrator uses a shared informer + workqueue pattern, not raw `watch` per Job.
|
||||
|
||||
---
|
||||
|
||||
## V1 Scope (Ships)
|
||||
|
||||
- ✅ `KubernetesExecutionTarget` kind in `executionTarget`
|
||||
- ✅ `@paperclipai/execution-target-kubernetes` package — orchestrator + driver
|
||||
- ✅ `ClusterConnection` model + storage + UI form
|
||||
- ✅ Namespace-per-company provisioning with all isolation primitives (RBAC, ResourceQuota, NetworkPolicy, PSS labels, image pull secret)
|
||||
- ✅ Job-per-run with PVC-per-agent + init container workspace strategy
|
||||
- ✅ Per-Job ephemeral Secret materialization with redaction layer
|
||||
- ✅ Bootstrap-token → run-JWT auth path (extends cursor-cloud-shared route)
|
||||
- ✅ K8s log watch + structured events callback + K8s Events forwarding
|
||||
- ✅ Cancellation, TTL cleanup, OwnerReference GC
|
||||
- ✅ Cilium auto-detection + CNP variant
|
||||
- ✅ Paperclip-maintained agent runtime image family (multi-arch, signed)
|
||||
- ✅ Per-namespace image pull credentials
|
||||
- ✅ Operator UI: cluster connection list, per-cluster health, per-tenant quota dashboard
|
||||
- ✅ Failure-mode error codes with remediation hints
|
||||
- ✅ Audit log integration
|
||||
- ✅ Test suite (unit + kind integration + security gate)
|
||||
- ✅ Documentation: quickstart, security model, multi-tenant onboarding playbook
|
||||
- ✅ Workspace-strategy refactor: extract shared library used by init container
|
||||
|
||||
## V2 (Designed-for, Deferred)
|
||||
|
||||
- 🟡 BYO cluster per company (today: per-instance only).
|
||||
- 🟡 Pod-per-agent mode (StatefulSet + KEDA scale-to-zero) as a per-adapter knob.
|
||||
- 🟡 External Secrets Operator integration via `SecretMaterializer` interface.
|
||||
- 🟡 VolumeSnapshot-based agent cloning.
|
||||
- 🟡 `PaperclipAgentRun` CRD + reconciliation operator (only if needed).
|
||||
- 🟡 Helm chart for the Paperclip control plane itself.
|
||||
- 🟡 Fine-grained image attestation (cosign verify in admission).
|
||||
- 🟡 IPv6 dual-stack pod support (V1 deny-list is IPv4-shaped).
|
||||
- 🟡 Cross-cluster TokenReview second-factor (needs identity federation).
|
||||
|
||||
---
|
||||
|
||||
## Risks & Open Questions
|
||||
|
||||
1. **Workspace strategy refactor scope.** Lifting workspace-strategy code out of `server/` into a shared library is a real refactor. The implementation plan must scope it precisely to avoid creep.
|
||||
2. **PVC zonal pinning UX.** First-zone-binds-forever is correct but surprising. Cluster-connection setup must explicitly call this out, with regional StorageClass guidance front-and-center.
|
||||
3. **`executionTarget` plumbing audit.** This spec assumes every existing adapter (`claude_local`, `codex_local`, etc.) plumbs `executionTarget` correctly through to its execute path. The implementation plan must include an audit and any plumbing fixes.
|
||||
4. **Resource defaults for `claude_local` in a Pod.** Long sessions can spike memory significantly. Need empirical numbers (50 representative agents in `kind`, capture p99 memory) before locking LimitRange defaults. Defaults in this spec are a starting point.
|
||||
5. **Cross-cluster TokenReview.** The V1.5 second-factor on `/api/agent-auth/exchange` needs identity federation between clusters. Documented as V2.
|
||||
6. **`registerExecutionTargetDriver()` doesn't exist yet.** The platform-module registry surface is extended in this spec; the implementation plan should add the registry as a small explicit step before plugging in the k8s driver.
|
||||
7. **Adapter `networkRequirements` field doesn't exist on `ServerAdapterModule` yet.** Adding `networkRequirements?: { allowFqdns?: string[] }` to the adapter contract is a small addition the implementation plan must include.
|
||||
8. **Agent shim binary.** `paperclip-agent-shim` is new code. Scope, language (Go for static binary preferred), and packaging into the runtime image must be designed in the implementation plan.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Decision Log
|
||||
|
||||
| Decision | Chosen | Rejected alternative(s) | Reason |
|
||||
|---|---|---|---|
|
||||
| Extension shape | New `kubernetes` kind on `AdapterExecutionTarget` | New `kubernetes_pod` adapter; CRD/operator | Highest leverage; every adapter inherits k8s; smallest blast radius |
|
||||
| Tenant boundary | Namespace per company | Cluster per company; label-based isolation | K8s-native; free isolation primitives; cluster-per-company doesn't scale |
|
||||
| Workload granularity | Job-per-run + PVC-per-agent | Pod-per-company (shared); pod-per-agent (default); pod-per-run (ephemeral PVC) | Strict isolation per run + warm workspaces; pod-per-company has poor isolation and doesn't actually save cluster cost |
|
||||
| Topology | Hybrid (in-cluster + cross-cluster) with bootstrap auth | Same-cluster only; cross-cluster only | One auth path serves both deployments |
|
||||
| Orchestration runtime | Imperative `@kubernetes/client-node` from server | CRD + Go operator; CRD + TS operator | Smallest moving parts; reuse k8s primitives; runs are ephemeral |
|
||||
| Workspace bootstrap | Init container running existing strategy | Server pre-populates via `kubectl cp`; one-shot prepare Job | Reuses strategy code; same FS layout as local; scales cross-cluster |
|
||||
| Namespace naming | `paperclip-{companySlug}` (operator-friendly) | `paperclip-{shortHash}` (collision-safe by construction) | kubectl debuggability; immutable label remains canonical |
|
||||
| FQDN egress control | Cilium auto-detected; vanilla NetworkPolicy as floor | FQDN-required (Cilium hard dep); egress proxy required | Works with any CNI; tightens when Cilium is present; floor blocks RFC1918 + link-local |
|
||||
| Secret injection | Native k8s Secret per Job, OwnerRef-GC'd, mounted as files | ExternalSecrets Operator (V2); CSI Secret Store (V2) | Simplest correct V1; abstraction in place for V2 drivers |
|
||||
| Retry semantics | Owned by Paperclip (`backoffLimit: 0`) | Owned by k8s (`backoffLimit: 6`); shared | Paperclip already has `AdapterExecutionErrorFamily` + `retryNotBefore`; double-retry breaks billing/audit |
|
||||
|
||||
---
|
||||
|
||||
*Spec ends.*
|
||||
Reference in new issue
Block a user