mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 08:22:20 +02:00
9ace548fd21536f83ca5dd8181bf978bc2a723ea
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
5858ccb981 |
feat: make in-app features cloud-aware (#10850)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use the same board application in self-hosted and Paperclip Cloud deployments. > - A Cloud tenant contains one company, so an in-app company switch does not change the active Cloud stack. > - Cloud operators need the sidebar and company surfaces to use the signed-in user's stack portfolio. > - The server must derive Cloud identity and links from trusted instance context instead of client input. > - This pull request adds canonical Cloud context, a trusted stack portfolio proxy, and Cloud-aware navigation. > - The benefit is consistent stack switching on Cloud while self-hosted company behavior stays unchanged. ## Linked Issues or Issue Description **Subsystem affected** Cross-cutting: server REST routes and the React board UI. **Problem or motivation** A Cloud-managed instance contains one company. The existing company switcher could only switch records inside that tenant. It could not move the operator to another Cloud stack. The existing header also gave long organization names too little width. **Proposed solution** Expose a canonical public Cloud context in health data. Add a trusted server proxy for the current user's stack portfolio. Use that data in the board UI to switch stacks with top-level navigation. Keep the existing company behavior on self-hosted instances. Move search into the navigation and keep long organization names inside the sidebar panel. **Alternatives considered** An in-app `/stacks` route was rejected because Cloud tenant hosts reserve that path and stack selection must wake or authenticate another tenant. Client-supplied user identity was rejected because the server can derive the trusted Cloud actor. **Roadmap alignment** This change advances the Cloud deployments milestone. It keeps the product local-first and Cloud-ready without changing the self-hosted mental model. ## What Changed - Added canonical Cloud instance context and public health metadata. - Added a Cloud-only stack portfolio proxy with trusted actor forwarding and per-user caching. - Prevented normal company creation on Cloud-managed instances. - Switched the sidebar and Companies page from company actions to stack actions on Cloud. - Added full-page stack navigation and Cloud create-stack links. - Moved search into the sidebar navigation so the organization name keeps more width. - Added truncation and hover recovery for long organization and stack names. - Added server and UI regression coverage for Cloud and self-hosted behavior. - Updated the implementation specification for the Cloud contracts. ## Verification - `node scripts/check-token-gates.mjs` passed. All three token gates are clean. - `pnpm --dir server exec vitest run src/__tests__/health.test.ts src/__tests__/cloud-instance.test.ts src/__tests__/cloud-routes.test.ts src/__tests__/company-cloud-floor.test.ts src/__tests__/company-portability-routes.test.ts` passed: 5 files and 66 tests. - `pnpm --dir ui exec vitest run src/components/SidebarCompanyMenu.test.tsx` passed: 1 file and 11 tests. - Pre-PR QA report `7da87ca7` passed all 8 acceptance criteria with real HTTP route factories and real Chromium screenshots in Cloud and self-hosted modes. - Security reviews passed for the canonical Cloud context and stack portfolio proxy. ## Risks - Cloud stack switching depends on the configured Cloud application and tenant portfolio URLs. - The new health `cloud` block is public by design, but it contains only canonical public instance metadata. - The stack proxy fails closed on self-hosted instances and derives the user identity from the trusted actor. - Self-hosted navigation and company creation retain their existing paths and behavior. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, model `gpt-5`. The run used reasoning, repository tools, shell execution, and GitHub integration. The deployment did not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
f51cba33fa |
fix(server): keep environment secret bindings consistent when re-pointing config secrets (#10576)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents can run inside environments (SSH boxes, sandbox providers); a sandbox environment's config can reference stored company secrets (for example a provider API key) through `format: "secret-ref"` fields > - Environments are instance-scoped and shared by every company on an instance, but `company_secret_bindings` rows are company-scoped, and the environment routes synced config-derived bindings under one guessed "context company" resolved from the environment's existing bindings > - When a save re-pointed a secret-ref field at a secret owned by a different company, the binding sync threw after the config row had already been persisted: the config referenced the new secret, the binding still pointed at the old one, every later lease acquisition failed with `Secret is not bound to environment:<id> at apiKey`, and the stale cross-company binding made every later save fail with a company-context conflict — with no route-level way to recover > - This pull request makes config-derived bindings follow the company that owns each referenced secret, and makes the environment write and its binding syncs atomic > - The benefit is that environment saves can no longer strand an environment in a half-updated state that breaks all of its runs ## Linked Issues or Issue Description Refs #10577 (companion UX change: the editor state that nudges operators into this sequence). **What happened?** Saving an environment whose secret-ref config field points at a secret owned by a different company than the environment's existing binding partially applied: the config row updated, the binding sync failed server-side, and the environment was left referencing a secret it has no binding for. Every run that leased the environment then failed with `lease_acquire_failed: ... Secret is not bound to environment:<id> at apiKey`, and every later save of the environment returned 409 `Environment secret bindings already use a different company context.` — with no route-level way to recover. **Steps to reproduce** 1. On an instance with two companies, create a sandbox environment from company A with a picker-bound API-key secret owned by A (the binding lands in A). 2. From company B, create a new secret and re-point the environment's API-key field at it, then save. 3. The save persists the config but the binding sync throws, so no binding for B's secret exists. 4. Run any agent that uses the environment, or try to save the environment again. **Expected behavior** The save either fully applies (config and bindings consistent) or fully fails. Re-pointing a config secret ref to a secret owned by another company moves the binding with the secret. **Paperclip version** Reproduced on current `master` (also present on recent release images). **Deployment mode** Multi-company server deployment (any mode with more than one company on the instance). ## What Changed - New `secretService.replaceSecretRefsForInstanceTarget`: writes each config-derived binding under the company that owns the referenced secret, replaces all non-`env.*` bindings of the target across every company, and validates every ref (secret exists, not deleted, config-path and projection-class rules) before any row is written. `env.*` env-var bindings stay company-scoped and untouched. - The environment create and update routes now run the environment write and its binding syncs inside one `db.transaction`, threading the transaction through new optional executor seams on `environmentService.create/update` and the existing `SecretBindingDb` seam pattern, so an invalid ref rolls the whole save back instead of leaving a half-updated environment. - `resolveEnvironmentSecretContextCompanyId` no longer lets existing bindings veto the caller's context (the 409s above); it now only picks where new raw-pasted secrets are created and how env-var bindings and probes resolve: explicit route/query company first, then the single company the bindings live in, then the actor's company. ## Verification - `cd server && pnpm vitest run src/__tests__/environment-routes.test.ts src/__tests__/environment-instance-routes.test.ts src/__tests__/secrets-service.test.ts src/__tests__/environment-custom-image-routes.test.ts` (165 tests, includes new coverage below) - New embedded-Postgres tests prove: a re-point moves the binding to the new secret's company and deletes the stale row; refs across several companies each bind under their own secret's company; an unknown secret ref rejects without touching existing bindings; `env.*` rows survive config-ref replacement. - New route tests prove: a cross-company re-point that previously 409'd now saves, with the update and binding replacement on the same transaction executor; a failing ref surfaces as 422. - `cd server && pnpm run typecheck` ## Risks - Behavioral shift: environment saves no longer 409 on a company-context mismatch between the caller and existing bindings; bindings follow the referenced secret's company instead. Environment routes are instance-admin gated, and instance admins already had access to every company's secrets by passing the company explicitly, so this removes an ordering trap rather than widening access. - Runtime lease resolution is unchanged: a run still resolves environment secrets under the run's own company, so an environment referencing company B's secret still only leases for company B runs (fail-closed as before). - The delete route's per-company binding cleanup is unchanged. ## Model Used Claude Fable 5 (`claude-fable-5`, Anthropic) via Claude Code — extended thinking, agentic tool use (file edits, vitest/tsc runs). No other models involved. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Devin FoleyandPaperclip
|
c274f10abc |
feat(server): computed owner instance-admin elevation for cloud-managed instances, behind platform floors (#10343)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud-managed instances authenticate tenant users through a trusted-header path (`resolveCloudTenantActor`) that deliberately never grants `instance_admin`, so every tenant user is company-scoped > - On a dedicated (single-owner) managed instance that leaves the paying owner unable to administer their own instance: instance settings, the environments admin surface, and the custom sandbox image flow are all instance-admin gated (the environments UI can't even show the provider/image of the platform sandbox because the restricted read view blanks `config` entirely) > - Re-granting the old blanket `instance_user_roles` row would repeat the mistake the shared-pool hardening fixed: DB rows go stale, resurrect via restores, and elevate through every auth path > - This pull request elevates only the stack `owner`, computed per request at the trusted-header boundary behind a new managed-tier feature flag, and ships that elevation together with code floors on the platform-owned surfaces an instance admin must not control on a managed instance > - The benefit is that dedicated-stack owners can administer their own instance while platform credentials, execution policy, backups, and runtime code-install stay platform-owned, and self-hosted behavior is unchanged ## Linked Issues or Issue Description No public GitHub issue exists for this change; the underlying issue is described here following the feature-request template. ### Problem or motivation - On cloud-managed instances, tenant users resolved from trusted headers are always company-scoped. For dedicated instances with a single paying owner, the owner cannot reach any instance-admin surface of their own instance (instance settings, environments administration, custom image setup), and the restricted environment read view hides even structural fields like the sandbox provider and image. - The previous hardening intentionally removed blanket elevation (and purges stale `instance_user_roles` rows on every trusted-header authentication). That protection must not regress for shared multi-tenant pools. ### Proposed solution - Owner-only, computed, flag-gated elevation plus code floors on platform-owned surfaces, in one PR so the elevation can never ship without the floors. ### Alternatives considered - Re-inserting an `instance_user_roles` row for owners (the pre-hardening model): rejected — DB rows go stale, survive restores, and elevate through every auth path; #7525 removed exactly this. - Widening only the environments read view without any elevation: rejected — it fixes one screen but still leaves a dedicated-instance owner unable to administer instance settings or custom images. - Elevating additional stack roles (`member`/`admin`/`support`): rejected — only the owner has an ownership claim over the whole instance; other roles stay company-scoped. ### Roadmap alignment - Extends the shipped "Cloud deployments" roadmap work (multi-tenant isolation, company-scoped cloud tenants, managed-instance bootstrap) without overlapping planned core items, and leaves self-hosted behavior unchanged. ## What Changed - **New feature key** `enableOwnerInstanceAdmin` (`packages/shared`): boolean flag in `instanceExperimentalSettingsSchema`, catalog tier `managed`, `cloudDefault: true`, `selfHostedDefault: false`. Inert on self-hosted instances — the elevation path only exists behind the cloud tenant trust token. - **Computed elevation** (`server/src/middleware/auth.ts`): `resolveCloudTenantActor` now returns `isInstanceAdmin: true` only when the trusted-header stack role is `owner` **and** the flag is enabled. The flag is resolved through the instance-settings service so the managed-config overlay applies (the control plane can disable elevation fleet-wide without touching tenant databases; a DB row edit or restore cannot resurrect it). Resolution fails closed on settings read errors. The `instance_user_roles` never-insert and the per-request stale-row purge are byte-identical. `member`/`admin`/`support` stack roles stay company-scoped. - **Authorization guard split** (`server/src/services/authorization.ts`): the blanket-allow now trusts the actor's *computed* `isInstanceAdmin` flag (only the attested resolver can set it for `cloud_tenant` actors) while keeping the `instance_user_roles` DB lookup excluded for `cloud_tenant` — a stale or hand-inserted role row still elevates nothing. - **Floor F1 — platform environment credentials** (`server/src/routes/environments.ts`): on cloud-managed instances, platform-provisioned environment rows (`managedByPaperclip` marker, plus the legacy managed-Kubernetes marker) use a single floored view for every reader on all environment routes (list, get, create, update, delete responses): `envVars` are never echoed and credential-shaped `config` keys (reusing the managed-config `SECRET_LIKE_CONFIG_KEY_PATTERN`) are dropped — for **all** actors including instance admins — while structural config (provider, image, template, region, …) and the managed markers stay visible. This also fixes the environments UI for managed sandboxes, which previously lost the provider/image entirely in the restricted view. The floor also covers writes: `PATCH /environments/:id` and `DELETE /environments/:id` on a platform-provisioned row are rejected (403, `environment_platform_managed`) for every actor including instance admins, and the guard binds to the persisted row's markers so a patch cannot strip the managed marker to lift the floor. The one recovery path is a metadata-only PATCH that solely clears the marker keys (null/false), for rows stamped through the old unrestricted API before the markers became reserved — and it never applies to a row whose slot markers are live platform state: the single local row (`environments_local_driver_idx`), which `ensureLocalEnvironment` adopts and stamps on cloud-managed instances from every caller (company creation, the heartbeat, run orchestration), and the single marked sandbox row (`environments_managed_sandbox_idx`) while a managed-sandbox bootstrap path is configured (managed-config `environments` section or `PAPERCLIP_EXECUTION_MODE=kubernetes`) and the provisioner therefore adopts and refreshes it on every boot. Clearing a live slot row's markers would let the next write reclassify it as tenant-managed and bypass the floor; conversely, when no sandbox provisioning path is configured the platform holds no claim on any sandbox row, so a platform marker there is stale by definition and the recovery patch applies. Every marker outside a live slot is clearable, so no legacy row is ever locked permanently. Custom-image setup and probes on the platform sandbox stay available to instance admins — those are the owner-facing flows this elevation exists for. The marker keys themselves are reserved: client create/update payloads that set `managedByPaperclip` or `managedKubernetesSandbox` are rejected (422, `environment_platform_marker_reserved`) on cloud-managed instances, so a tenant row can never be stamped platform-provisioned through the API and self-locked behind the write floor (the provisioner writes markers at the service layer, not through these routes). Tenant-created environments are otherwise unaffected. - **Floor F2 — executionMode** (`server/src/routes/instance-settings.ts`): on cloud-managed instances, `PATCH /instance/settings/general` rejects writes that would change `executionMode` (403, `execution_mode_platform_managed`). Same-value echoes pass so settings forms that submit the full general-settings object keep working. The boot-time execution-policy bootstrap path is untouched (it calls the service directly). - **Floor F3 — manual database backups** (`server/src/routes/instance-database-backups.ts`): floored off on cloud-managed instances (403, `database_backups_platform_managed`); backups are platform-owned there, and the result would also echo a server-side filesystem path. - **Floor F4 — adapter code install** (`server/src/routes/adapters.ts`): `POST /adapters/install` and `POST /adapters/:type/reinstall` are floored off on cloud-managed instances (403, `adapter_install_platform_managed`). Adapter packages execute in the server process, so a runtime install would let an instance admin read the platform trust anchors out of the process environment. This mirrors the existing bundled-only plugin install floor; adapter code on managed instances comes bundled with the platform image. ## Instance-admin surface audit Before widening who can hold `isInstanceAdmin`, every instance-admin-gated surface in `server/src` was enumerated and reviewed for whether its response or side effects could echo process environment values or platform credentials (tenant trust token, JWT signing keys, database connection strings, provider API keys): 29 distinct gate definitions covering ~90+ call sites, in four groups — sole instance-admin gates (12), instance-admin-or-company-permission gates (10), response-shaping/scope-widening sites (6), and the central `allow_instance_admin` short-circuit in the authorization service (58 `decide()` call sites). Findings and dispositions: - **Environment read/write responses** exposed platform sandbox `envVars`/credential-shaped config to instance admins → closed by floor F1. - **Manual backup trigger** echoed a server filesystem path and triggers a platform-owned operation → closed by floor F3. - **Adapter install/reinstall** loads externally fetched code into the server process (indirect, complete env exposure) → closed by floor F4. The sibling plugin-install path already had a bundled-only floor on managed instances and needed no change. - **Token-minting surfaces** (gateway tokens, custom-image terminal/connection tokens) mint credentials scoped to the instance's own resources, not platform trust anchors → acceptable for an owner-admin of a dedicated instance; unchanged. - All remaining gated surfaces return ordinary instance-scoped business data; none echo `process.env` or platform secrets directly. OAuth client secrets are referenced by env-var *name* only; SSH private keys are stored as secret refs before persistence and are not echoed. Operational note for managed platforms: this model assumes the process environment of a managed instance holds only that instance's own credentials. Platform operators should keep provider credentials per-instance (never fleet-shared) since an instance admin ultimately controls in-process code on their own instance. ## Verification - `pnpm vitest run server/src/middleware/cloud-tenant-actor.test.ts` — resolver matrix: owner × flag on/off, flag via managed overlay (on-over-DB-off and off-over-DB-on), member/admin/support × flag on, no-token self-hosted, fail-closed settings read, purge still runs and no role row is ever inserted (14 tests). - `pnpm vitest run server/src/__tests__/authorization-service.test.ts` — computed flag elevates a `cloud_tenant` actor; a stale `instance_user_roles` row still never does; `session` actors unchanged (full suite, embedded Postgres). - `pnpm vitest run server/src/__tests__/environment-routes.test.ts` — F1: no secret echo to admins on get/list, structural config visible to restricted readers, platform-row PATCH/DELETE rejected for admins (including a marker-stripping patch), marker-clear recovery allowed for stale legacy rows and for a marked sandbox row when no provisioning path is configured, but refused on the managed local row and on the sandbox slot row under a managed-config `environments` entry or the forced kubernetes execution mode, client marker-stamping creates/patches rejected, tenant rows still readable and writable, self-hosted read+write regression (60 tests). - `pnpm vitest run server/src/__tests__/environment-service.test.ts` — `ensureLocalEnvironment` adopts a pre-existing local row on cloud-managed instances (marker stamped, other metadata preserved, idempotent — no rewrite on re-ensure) and leaves self-hosted rows untouched (22 tests, embedded Postgres). - `pnpm vitest run server/src/__tests__/instance-settings-routes.test.ts server/src/__tests__/instance-database-backups-routes.test.ts` — F2 change-vs-echo matrix incl. self-hosted regression; F3 floor for both admin shapes (32 tests). - `pnpm vitest run server/src/__tests__/adapter-routes-authz.test.ts` — F4 floor; self-hosted install/reinstall behavior unchanged (existing cases). - `pnpm vitest run server/src/__tests__/first-admin-claim.test.ts server/src/__tests__/bootstrap-claim-routes.test.ts server/src/__tests__/managed-config.test.ts server/src/__tests__/health.test.ts server/src/__tests__/instance-settings-managed-overlay.test.ts server/src/services/managed-environments.test.ts server/src/services/execution-policy-bootstrap.test.ts` — first-admin bootstrap gate and managed-config behavior unchanged (91 tests). - `pnpm vitest run packages/shared/src/feature-catalog.test.ts` — catalog/schema sync tests cover the new key (selfHostedDefault must equal the schema default). - `pnpm run typecheck` — all 31 workspace projects clean. ## Risks - Self-hosted behavior is unchanged: every floor binds to `isCloudManagedInstance()` (tenant trust token present), the new flag defaults off with no elevation path, and regression tests pin the self-hosted branches. - The elevation is fail-closed and stateless: turning the flag off (managed overlay or DB) de-elevates on the next request; there is no role row to clean up and restores cannot resurrect elevation. - On a cloud-managed instance a pre-existing unmarked local row is adopted (stamped `managedByPaperclip`) by the next ensure and becomes platform-owned — the intended managed-product semantic: the platform owns the single local slot. Self-hosted instances are untouched. - F1 widens restricted readers' view of platform-provisioned rows from fully blanked `config`/`metadata` to structural-only `config` plus markers. Platform-delivered config is guaranteed secret-free by the managed-config contract (secret-shaped keys fail startup), and the floor re-drops secret-shaped keys defensively. - One extra instance-settings read per trusted-header request for owner-role actors (the resolver already performs several queries per request). ## Model Used Claude Fable 5 (Anthropic) — model id `claude-fable-5`, extended thinking enabled, agentic tool use via Claude Code; read-only explore subagents on the same model were used for the surface audit sweep. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
273315a4d0 |
feat(server): provision managed sandbox environments from the managed config (#10324)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted/managed deployments configure instances entirely from the control plane: `PAPERCLIP_MANAGED_CONFIG` already delivers feature flags and `plugins.autoInstall` (bundled sandbox provider plugins), parsed fail-closed at boot > - A plugin alone is not usable for execution: runs need an instance-level `driver: "sandbox"` environment row pointing at the provider, and today only Kubernetes has a boot path for that (`PAPERCLIP_EXECUTION_MODE` → `ensureKubernetesEnvironment`); every other provider requires a manual product-API call the control plane cannot make on a managed instance > - Adding one `ensureXxxEnvironment` per provider would multiply near-identical boot hooks and env-var surfaces > - This pull request generalizes the existing Kubernetes machinery: the managed-config document gains an optional `environments` section that declares a sandbox environment for any bundled provider, ensured idempotently at boot by a provider-agnostic service function (the Kubernetes hook becomes a thin wrapper over it) > - The benefit is that a managed fleet can provision any sandbox provider (Daytona, Modal, E2B, …) purely from configuration — no per-provider code, no manual API calls, no secrets in the document — while self-hosted behavior is untouched ## Linked Issues or Issue Description No public issue exists. Refs #10157 (the cloud image variant that bundles sandbox provider plugins — this PR is the configuration half that makes an installed provider usable). **Problem (feature-request shape):** on a managed instance the control plane can auto-install a bundled sandbox provider plugin via `PAPERCLIP_MANAGED_CONFIG.plugins.autoInstall`, but cannot create the environment row that makes the provider schedulable. The only boot-time environment provisioning is Kubernetes-specific (`PAPERCLIP_EXECUTION_MODE=kubernetes` + `PAPERCLIP_K8S_*`). A generic, config-driven path is needed so any bundled provider can be provisioned without per-plugin code or manual API calls. ## What Changed - `server/src/services/managed-config.ts`: optional `environments` top-level section — `[{ name, description?, provider, config? }]` — validated fail-closed: unknown keys, more than one entry (the DB permits exactly one Paperclip-managed sandbox row, `environments_managed_sandbox_idx`), a `provider` not present in `plugins.autoInstall`, `config.provider`, or secret-looking config keys at any depth (`api_key`/`token`/`secret`/`password`/`credential`) all refuse startup. Absent section ⇒ `environments: []`, so pre-section documents keep booting newer builds. - `server/src/services/environments.ts`: new provider-agnostic `ensureManagedSandboxEnvironment({ name, description?, provider, config?, extraMetadata? })` — idempotently owns the single managed sandbox row: refreshes name/description/config each call, adopts the slot across provider switches (dropping the stale `managedKubernetesSandbox` marker), adopts a same-name unmanaged sandbox row (stamping it managed) instead of colliding on `environments_name_idx` every boot, and falls back to keeping the current name if the desired name belongs to a different row. `ensureKubernetesEnvironment` is now a thin wrapper that pins `provider: "kubernetes"` and stamps the legacy marker. - `server/src/services/managed-environments.ts` (new): `applyManagedEnvironments` boot step — no-op for self-hosted/empty; throws (fail startup) when `PAPERCLIP_EXECUTION_MODE` is also set, since both would own the same managed sandbox row; otherwise ensures each declared environment fail-safe per entry (log + continue boot, matching bundled-plugin provisioning posture). - `server/src/index.ts`: runs the new boot step right after the execution-policy bootstrap, before the heartbeat resumes queued runs. - `server/src/services/index.ts`: exports `applyManagedEnvironments` and `ManagedEnvironmentSpec`. - Secrets stay out of the document by construction: provider credentials reach managed instances only as process env vars (each provider's documented fallback, e.g. `DAYTONA_API_KEY` for the Daytona plugin). ## Verification ```sh cd server pnpm exec tsc --noEmit -p tsconfig.json pnpm exec vitest run \ src/__tests__/managed-config.test.ts \ src/services/managed-environments.test.ts \ src/services/execution-policy-bootstrap.test.ts \ src/__tests__/environment-service.test.ts \ src/__tests__/environment-instance-routes.test.ts \ src/__tests__/environment-routes.test.ts \ src/__tests__/plugin-install-guard.test.ts \ src/__tests__/environment-execution-target.test.ts \ src/__tests__/instance-settings-managed-overlay.test.ts \ src/__tests__/bundled-plugins.test.ts ``` All pass locally (typecheck clean; environment-service suite runs against embedded Postgres and exercises the refactored Kubernetes wrapper plus the new generic ensure: create/refresh, provider switch, unmanaged-row adoption, name-conflict fallback). New tests cover the parser (12 cases incl. secret-key rejection at depth) and the boot step (no-op, mutual exclusion, pass-through, fail-safe). ## Risks - **Self-hosted: none intended.** Without `PAPERCLIP_MANAGED_CONFIG` nothing new executes; the `PAPERCLIP_EXECUTION_MODE=kubernetes` path is regression-covered by the existing bootstrap/service/route suites (all green). - **Behavioral shift in `ensureKubernetesEnvironment` (deliberate):** it now also refreshes `name`/`description` to their managed defaults each boot (desired-state semantics, same as config today) and adopts a `managedByPaperclip` sandbox row that lacks the Kubernetes marker — previously that state made the ensure throw every boot. - **New startup failure modes are all explicit misconfigurations** (malformed section, provider not auto-installed, secret in config, execution-mode conflict) and fail with precise errors; DB-side ensure failures never block boot (fail-safe per entry, logged). - No migrations; no API surface changes. ## Model Used Claude Fable 5 (Anthropic, model ID `claude-fable-5`) with extended thinking and tool use, driving the change end-to-end inside a Claude Code / agent-harness session (code, tests, and verification runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (the managed-config module header is the contract doc) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
|
719da5f9b5 |
Harden environment deletion and expose delete blast radius (#9250)
## Thinking Path > - Paperclip manages AI agents that each have an associated execution environment (local, Kubernetes, etc.) > - Instance administrators can create and delete environments; currently the DELETE endpoint has no protection against deleting managed or in-use environments > - Deleting the managed local environment or the instance-default environment would break all agents using those environments with no path to recovery > - The endpoint also suffered a TOCTOU race: a check-then-delete pattern allowed the managed-local or default guard to pass if the environment's role changed between the read and the delete > - This pull request adds a blast-radius read endpoint so admins can preview impact, hard-blocks the dangerous deletes atomically, cleans up all dependent references after a valid delete, and fixes a concurrent creation race in ensureLocalEnvironment ## Linked Issues or Issue Description Fixes #9251 ## What Changed - **New endpoint** `GET /api/environments/:id/delete-blast-radius` (instance-admin gated): returns reference counts (agent defaults, workspace selections, issue selections, project selections, secret bindings, active leases, active setup sessions) and blocking reasons — no config, env-var values, or secret data returned. - **Atomic delete guard** `environmentService.removeIfDeletable(id)`: performs the DELETE with an inline `WHERE driver != 'local' AND NOT EXISTS (instanceSettings where defaultEnvironmentId = id)` predicate, eliminating the TOCTOU race between the app-level check and the DB write. - **Route hardening**: `DELETE /environments/:id` now calls `getDeleteBlastRadius` first (app-level check + logging), then calls `removeIfDeletable` (atomic guard). If the atomic guard returns null the route fetches a fresh blast-radius snapshot and rejects with a 409 Conflict carrying `deleteBlockedReasons`. - **Reference cleanup on valid delete**: after a successful delete, the route clears environment selections on all company execution workspaces, issues, and projects; syncs env-var secret bindings to `{}` (removing bindings for the deleted environment); syncs config secret refs to `[]` for the environment target; and removes the SSH private-key secret if one was stored. - **Race fix in `ensureLocalEnvironment`**: the insert-or-nothing path now catches a `environments_name_idx` unique-constraint violation and falls through to the existing SELECT, treating the name conflict as idempotent. - **Shared types**: `EnvironmentDeleteBlastRadius` and `EnvironmentDeleteBlockedReason` exported from `@paperclipai/shared`. - **OpenAPI**: registers the new blast-radius endpoint; updates the delete-environment response schema to document 403/404/409. - **Tests**: 56 existing environment-route and service tests continue to pass; new service-level regression tests assert the atomic guard rejects `local`-driver environments and instance-default environments and succeeds for deletable ones. ## Verification ``` corepack pnpm exec vitest run \ server/src/__tests__/environment-routes.test.ts \ server/src/__tests__/environment-service.test.ts # 56 tests, all passing corepack pnpm --filter @paperclipai/shared typecheck node scripts/ensure-plugin-build-deps.mjs cd server && ../node_modules/.bin/tsc --noEmit ``` ## Risks - **Blast-radius endpoint auth**: guarded by `assertCanAccessInstanceEnvironments`, the same gate as the existing environment-list and delete routes. Non-admin callers receive 401/403 before any data is returned. - **Atomic guard may reject a delete that the app-level check passed**: this is intentional — it means the environment became protected between the read and the write. The caller receives a fresh blast-radius snapshot explaining why. - **Secret cleanup ordering**: cleanup runs after the atomic DELETE succeeds, in parallel across companies. If cleanup partially fails the environment row is already gone; partial-cleanup state is recoverable by re-running the sync operations. Risk: low — these are idempotent upsert/sync operations. - **ensureLocalEnvironment race fix**: swapping a unique-constraint error for an idempotent SELECT adds one extra query on the conflict path. This path is rare (only fires during concurrent boot) and is significantly safer than the previous behavior. - **No migration**: all changes are application-level; no schema changes required. ## Model Used - Provider: Anthropic - Model: claude-sonnet-4-6 (Claude Sonnet 4.6) - Context window: 200k tokens - Mode: agentic tool use via Paperclip agent system (Claude Code) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Priya Raman <priya.raman@paperclip.ing> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Harold Kim <harold.kim@paperclip.ing> |
||
|
Devin Foley
|
cd38c150b0 |
feat: reuse Daytona sandbox leases (#8513)
Add opt-in reusable Daytona sandbox lease support, including retryable pending cleanup handling.\n\nPR: https://github.com/paperclipai/paperclip/pull/8513 |
||
|
Devin Foley
|
547463d3a2 |
refactor(environments): make execution environments instance-scoped (#8375)
## Thinking Path > - Paperclip is the control plane for AI-agent companies, so execution environment selection has to stay inspectable and predictable across companies, agents, and runs. > - The environment subsystem decides where an agent heartbeat actually runs and how remote sandbox state is realized and restored. > - That subsystem previously mixed company-scoped environment catalogs with issue-level environment stamping, so a reassigned issue could keep executing in the previous assignee's sandbox. > - That behavior breaks the control-plane contract: changing the assignee should change the executing agent/environment path unless there is an explicit current override. > - Fixing it cleanly required more than a narrow patch; the environment model had to move to instance scope with a single inherited default and per-agent override semantics. > - This pull request rewires the schema, server/API surface, runtime resolution, and UI around that model, then adds regression coverage for cross-company inheritance and per-agent isolation. > - The benefit is that environment choice now follows the approved instance/agent configuration path instead of stale issue state, while shared environments only need to be configured once per instance. ## Linked Issues or Issue Description - No directly matching public GitHub issue or PR was found while searching for this refactor. ### What happened? Reassigning work between agents with different execution environments could keep running in the previous sandbox because environment choice was stamped onto the issue and outranked the current assignee. The same subsystem also forced environment catalogs to be duplicated per company even though the underlying execution environments were instance-wide resources. ### Expected behavior Execution should resolve through the current instance and agent configuration path, with one instance-scoped environment catalog, one instance default, optional per-agent override, and no stale issue-level environment authority surviving reassignment. ### Steps to reproduce 1. Configure two agents to use different execution environments. 2. Assign an issue to the first agent so the issue records execution state in that environment. 3. Reassign the same issue to the second agent and run another heartbeat. 4. Observe that the pre-fix runtime can still sync or execute in the original sandbox instead of the second agent's environment. ### Paperclip version or commit Current `master` before this PR. ### Deployment mode Self-hosted server. ### Installation method Built from source (`pnpm dev` / `pnpm build`). ### Agent adapter(s) involved - Claude Code - Not adapter-specific (core bug in environment authority / resolution) ### Database mode External Postgres. ### Access context Both board reassignment and agent heartbeats were involved. ## What Changed - Moved environments and their default selection contract to instance scope in DB/shared types, including the migration that dedupes legacy per-company environments and seeds the instance local default. - Reworked environment CRUD/auth flows to use instance-scoped APIs and added route/service coverage for instance-level environment management. - Changed runtime resolution to prefer `agent default -> instance default -> built-in local`, removed issue-level environment stamping from the active execution path, and isolated sandbox/plugin leases by `(executionWorkspaceId, agentId)`. - Added environment env-var runtime precedence so environment-provided values act as the baseline for agent execution. - Moved the environment UI into instance settings and updated agent configuration surfaces to reflect inherit/override behavior. - Added regression coverage for instance-default inheritance across companies and for the new runtime resolution behavior. - Fixed a rebase-only duplicate `enableTaskWatchdogs` flag regression in instance settings types/validators/services so the branch typechecks cleanly on current `master`. - Updated stale server tests so CI matches the shipped instance-scoped environment contract. ## Verification - `git diff --check` - `pnpm --filter @paperclipai/shared typecheck` - `pnpm --filter @paperclipai/db typecheck` - `pnpm exec vitest run server/src/__tests__/environment-runtime-driver-contract.test.ts server/src/__tests__/agent-permissions-routes.test.ts server/src/__tests__/environment-routes.test.ts server/src/__tests__/environment-instance-routes.test.ts server/src/__tests__/execution-workspace-policy.test.ts server/src/__tests__/heartbeat-plugin-environment.test.ts server/src/__tests__/instance-settings-routes.test.ts` ## Risks - The migration changes environment scope and dedupes existing rows, so installs with unusual legacy environment combinations should be reviewed carefully during upgrade. - Remote execution behavior now depends on instance-default inheritance semantics instead of issue-level stamping, so any remaining code paths that still assume issue-scoped environment authority would surface as follow-up bugs. - This PR includes both server/runtime behavior and UI relocation, so reviewers should watch for authorization edge cases around instance settings and environment management. > I checked [`ROADMAP.md`](ROADMAP.md). This work fits the existing Cloud / Sandbox agents direction as a bug-fix/refactor to current behavior, not a new parallel product surface. ## Model Used - OpenAI Codex coding agent in this Paperclip/Codex session; GPT-5-class tool-using model with code execution and shell access. The exact backend model ID is not exposed to the session runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Devin FoleyandPaperclip
|
f3e01c63bd |
fix(environments): partial unique index to dedup managed sandbox rows (#8247)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The `server/services/environments.ts` module lazily provisions a
managed Kubernetes sandbox environment for each company on first
heartbeat. Idempotency relies on `ensureKubernetesEnvironment` returning
the single managed row per company.
> - The `(company_id, driver)` unique index in the `environments` schema
is partial on `driver='local'` only, so two concurrent callers (e.g.
simultaneous first heartbeats from a freshly synced tenant) can both
insert a `driver='sandbox'` row before either sees the other.
> - The function tried to converge after the race by re-reading, picking
the oldest managed row as winner, and deleting the loser. Under
autocommit + read-committed, each post-insert SELECT is a fresh snapshot
— A may not see B's row, B may not see A's, so both pick their own and
neither deletes. Two rows survive.
> - The same race fired in CI as `expected 2 to be 1` at
`environment-service.test.ts:293`, gating multiple unrelated PRs on
retries.
> - This pull request encodes the operator-level invariant ("at most one
Paperclip-managed sandbox row per company") at the DB layer with a
partial unique index, then switches `ensureKubernetesEnvironment` to
`INSERT … ON CONFLICT DO NOTHING` keyed on that index. Losers re-read
the surviving row.
> - The benefit is the race is impossible by construction — no
application-side convergence loop, no test flake, and any future
`ensureXyzSandboxEnvironment` that sets `managedByPaperclip=true`
inherits the invariant for free.
## Linked Issues or Issue Description
Paperclip issue: PAPA-783 — implement managed-sandbox dedup fix (phase 2
of the approved plan on the parent flake-investigation issue).
This is the phase-2 fix for the flaky `environmentService > deduplicates
concurrent managed Kubernetes environment creation` test introduced by
`4ad94d0bd` (PR #7938). Failing CI runs since then on at least PRs
#7595, #8233, #8215, #8212. The plan was reviewed and approved on the
parent issue before implementation.
Closely related (not duplicates):
- PR #7938 — introduced the test and the in-process convergence loop
being replaced here.
- PR #7595, PR #8233, PR #8215, PR #8212 — downstream PRs affected by
the flake; one of them will be rebased onto this fix as the acceptance
gate.
## What Changed
- `packages/db/src/schema/environments.ts`: added
`environments_company_managed_sandbox_idx`, a partial unique index on
`(company_id) WHERE driver='sandbox' AND
(metadata->>'managedByPaperclip')::boolean = true`. The umbrella
`managedByPaperclip` predicate covers any current or future
Paperclip-managed sandbox flavor without needing a new index per
provider.
- `packages/db/src/migrations/0102_managed_sandbox_dedup_index.sql`:
one-shot dedup `DELETE` keeping the oldest managed-sandbox row per
`company_id` (scoped to `driver='sandbox' AND managedByPaperclip=true`),
`RAISE NOTICE` if any duplicates were removed, then `CREATE UNIQUE INDEX
IF NOT EXISTS environments_company_managed_sandbox_idx`. `CONCURRENTLY`
is omitted because the codebase's migration runner wraps each file in a
transaction (see `applyPendingMigrationsManually`); the table holds 1–3
rows per company, so the short lock is acceptable and consistent with
every other migration in the repo.
- `server/src/services/environments.ts`: `ensureKubernetesEnvironment`
now uses `INSERT … ON CONFLICT DO NOTHING` keyed on the new index. On
conflict it re-reads the surviving managed-sandbox row and returns it.
Drops the post-insert convergence (re-read by `createdAt ASC, id ASC`,
delete the loser) and the trailing comment that flagged "until a partial
unique index is added via migration" as the proper long-term fix.
- Unused `asc` import removed from
`server/src/services/environments.ts`.
## Verification
Local (matches the success criteria in the issue body):
```
$ cd server
$ passes=0; for i in $(seq 1 20); do
pnpm vitest run src/__tests__/environment-service.test.ts -t "deduplicates concurrent" \
&& passes=$((passes+1)) || break
done; echo "$passes/20"
20/20
$ passes=0; for i in $(seq 1 10); do
pnpm vitest run src/__tests__/environment-service.test.ts \
&& passes=$((passes+1)) || break
done; echo "$passes/10"
10/10
```
Adversarial fan-out stress (temporarily bumped `Array.from({ length: 8
}, …)` to `length: 32` on both dedup tests; reverted before commit):
```
$ # both dedup tests fan-out-of-32, 10 iterations
10/10
```
Two ensure paths exist in the parent plan, but only
`ensureKubernetesEnvironment` is on `master`.
`ensureManagedSandboxEnvironment` (referenced by the approved plan as
commit `dce9a9622`) lives only on an unmerged feature branch, not
master. The plan's helper-extraction and symmetric dedup test for that
path are deferred to whichever PR lands the second ensure path — it
inherits the same DB invariant by setting `managedByPaperclip=true`.
Discrepancy was flagged on the issue thread before implementation.
Typecheck:
```
$ pnpm -C server typecheck
ok
```
## Risks
Low risk.
- **Migration safety.** `IF NOT EXISTS` on the index makes the migration
idempotent. The dedup `DELETE` is bounded to rows matching the
managed-sandbox predicate; in production this should be a no-op (no race
has been reported in the wild — only in CI). On dev/CI DBs that already
accumulated duplicates, the migration emits a `NOTICE` reporting the
count.
- **No `CONCURRENTLY`.** The migration runner wraps each `.sql` file in
a transaction, which is incompatible with `CREATE INDEX CONCURRENTLY`.
The `environments` table holds 1–3 rows per company and the row count is
bounded by company count; the short ACCESS EXCLUSIVE during `CREATE
UNIQUE INDEX` is acceptable here and matches every other index migration
in the repo.
- **Predicate scope.** The partial index predicate matches exactly the
rows that `ensureKubernetesEnvironment` writes (`driver='sandbox'` with
`metadata.managedByPaperclip=true`). Tenant-created sandbox envs (via
`svc.create`) do not set this marker and are not covered — no false
positives, no surprise constraint violations on unrelated inserts.
## Model Used
Claude (Anthropic), `claude-opus-4-7`. Tool use: code edit + bash +
filesystem search; no extended-thinking mode.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
Jannes StubbemannandClaude Opus 4.8
|
4ad94d0bde |
feat(server): kubernetes execution integration for sandbox-provider plugins (stage 2/3) (#7938)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The execution subsystem runs those agents in environments (local, ssh, sandbox), and sandbox-provider plugins let an environment materialize per-run sandboxes > - Stage 1 (#5790) contributed a first-party Kubernetes sandbox-provider plugin, but the server core has no way to adopt it operationally: no per-run adapter selection, no way to force an instance onto sandboxed execution, no declarative adapter/model configuration, and the plugin must be installed by hand > - Without this, a multi-tenant or security-conscious deployment cannot guarantee that agent runs never execute on the host, and a single environment cannot serve agents with different harnesses > - This pull request adds the server + SDK integration: per-run adapterType on the lease protocol, an env-gated forced-Kubernetes execution policy with provisioning and a per-run allowlist guard, a declarative adapter registry and model list, in-cluster env passthrough for sandbox plugin workers, fail-safe auto-install of the bundled plugin, and the matching UI affordance > - The benefit is that sandbox-provider plugins become fully usable for Kubernetes execution: operators configure everything via environment variables and GitOps, while self-hosters who set none of the variables see exactly the behavior they have today ## Linked Issues or Issue Description Refs #5790 (stage 1 of 3: the Kubernetes sandbox-provider plugin package). No existing issue. Feature description: the server core lacks the integration seams to operate a sandbox-provider plugin as the mandatory execution path of an instance. This PR is stage 2 of 3 of the staged Kubernetes contribution; stage 3 will contribute the agent runtime images and their build pipeline. ## What Changed One line per piece: - `packages/plugins/sdk/protocol.ts`: optional `adapterType` on `PluginEnvironmentAcquireLeaseParams` so a provider can select the runtime image per run; existing providers simply ignore it - `server/services/environment-runtime.ts` + `environment-run-orchestrator.ts`: thread the agent's adapter type into both lease-acquiring drivers, including the heartbeat path (the two call sites have historically drifted, hence the pinned test) - `server/services/environments.ts`: `ensureKubernetesEnvironment` / `findKubernetesEnvironment`, an idempotent managed Kubernetes environment per company, identified by a metadata marker and refreshed (not recreated) on config change; `timeoutMs` rides on the config for slow cold-start leases - `server/services/execution-allowlist.ts`: pure (driver, provider, policy) -> allow/deny guard; `executionMode=kubernetes` only allows the kubernetes sandbox provider - `server/services/execution-policy-bootstrap.ts` + startup hook in `server/index.ts`: parse `PAPERCLIP_EXECUTION_MODE` / `PAPERCLIP_K8S_*`, persist `executionMode` into instance general settings, and provision the managed environment for every company; fails loud on misconfiguration - `server/services/heartbeat.ts`: when the policy forces Kubernetes, pin run selection to the managed environment (also overriding any persisted workspace environment id), refuse to fall back to local, and re-check the actually acquired environment against the allowlist as defense in depth - `server/services/adapter-registry-bootstrap.ts` + shared `AdapterRegistryEntry` type/validator: declarative `PAPERCLIP_ADAPTERS` registry (inline JSON or file) that reconciles adapter availability at startup and rides on the Kubernetes environment config - `server/services/adapter-models-env.ts` + `adapters/registry.ts`: `PAPERCLIP_ADAPTER_MODELS` lets an operator declare picker model lists the server cannot CLI-discover - `server/services/plugin-loader.ts`: pass `KUBERNETES_SERVICE_HOST/PORT(_HTTPS)` through to plugin workers that register environment drivers, so in-cluster API clients can be constructed; all other host env stays stripped - `server/app.ts`: fail-safe auto-install of the bundled kubernetes plugin at boot; no-ops when the bundle is absent and never blocks startup on error - `packages/shared` types/validators: `InstanceExecutionMode` on general settings (optional, strict schema) - `ui/lib/forced-kubernetes-environment.ts` + `AgentConfigForm`: when the policy is active, show a read-only Kubernetes environment instead of the environment picker and default new agents onto the managed environment - Tests for every new module plus the adapterType pin in `heartbeat-plugin-environment` and the managed-environment lifecycle in `environment-service` Everything is gated: with `PAPERCLIP_EXECUTION_MODE`, `PAPERCLIP_ADAPTERS`, and `PAPERCLIP_ADAPTER_MODELS` unset (and no bundled plugin present), every code path reduces to current behavior. The per-run `adapterType` is an optional SDK parameter that existing providers ignore. ## Verification - `cd server && npx tsc --noEmit`: clean (0 errors); `ui` typecheck also clean - Targeted suites all green (11 files, 90 tests): `npx vitest run server/src/__tests__/heartbeat-plugin-environment.test.ts server/src/__tests__/environment-service.test.ts server/src/__tests__/environment-runtime.test.ts server/src/__tests__/environment-run-orchestrator.test.ts server/src/__tests__/plugin-database.test.ts server/src/services/execution-policy-bootstrap.test.ts server/src/services/execution-allowlist.test.ts server/src/services/adapter-registry-bootstrap.test.ts server/src/services/adapter-registry-bootstrap.reconcile.test.ts server/src/services/adapter-models-env.test.ts packages/shared/src/validators/adapter-registry.test.ts` - `npx vitest run ui/src/components/AgentConfigForm.test.ts`: green (6 tests) - Full `npx vitest run server/src/__tests__`: 2323 passed, 1 skipped; the only failures (heartbeat-process-recovery pid-retry, workspace-runtime symbolic-ref/git tests) reproduce identically on pristine `master` in the same environment, so they are machine-environment issues unrelated to this change; `server-startup-feedback-export` needed its `services/index.js` mock extended with the new export and is green - This integration has been running in production on a hosted multi-tenant deployment, where it executes agent runs across five different harnesses through the stage 1 plugin ## Risks - Low for existing deployments: every behavior is env-gated and the defaults preserve current semantics; the auto-install block is wrapped fail-safe and skips silently when the plugin bundle is absent - `executionMode` is a new optional field on a strict zod schema; absent input normalizes exactly as before - The forced policy intentionally fails runs loudly (rather than falling back to local) when no managed Kubernetes environment exists; this only affects instances that explicitly set `PAPERCLIP_EXECUTION_MODE=kubernetes` ## Model Used Claude Opus 4.8 (claude-opus-4-8, 1M context), extended thinking, agentic tool use via Claude Code. ## UI screenshots The UI change is a new read-only "Execution" section in `AgentConfigForm`, shown only when the instance execution policy forces Kubernetes (`executionMode=kubernetes`); there is no "before" state for it (the section did not exist, and instances without the forced policy render the existing picker unchanged). Captured from the new Storybook stories added in this PR (`Product/Agent Management`): Managed Kubernetes environment present (read-only display, no local/SSH picker):  No managed environment available yet (warning notice, no silent local fallback):  ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Devin Foley
|
70679a3321 |
Add sandbox environment support (#4415)
## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies. > - The environment/runtime layer decides where agent work executes and how the control plane reaches those runtimes. > - Today Paperclip can run locally and over SSH, but sandboxed execution needs a first-class environment model instead of one-off adapter behavior. > - We also want sandbox providers to be pluggable so the core does not hardcode every provider implementation. > - This branch adds the Sandbox environment path, the provider contract, and a deterministic fake provider plugin. > - That required synchronized changes across shared contracts, plugin SDK surfaces, server runtime orchestration, and the UI environment/workspace flows. > - The result is that sandbox execution becomes a core control-plane capability while keeping provider implementations extensible and testable. ## What Changed - Added sandbox runtime support to the environment execution path, including runtime URL discovery, sandbox execution targeting, orchestration, and heartbeat integration. - Added plugin-provider support for sandbox environments so providers can be supplied via plugins instead of hardcoded server logic. - Added the fake sandbox provider plugin with deterministic behavior suitable for local and automated testing. - Updated shared types, validators, plugin protocol definitions, and SDK helpers to carry sandbox provider and workspace-runtime contracts across package boundaries. - Updated server routes and services so companies can create sandbox environments, select them for work, and execute work through the sandbox runtime path. - Updated the UI environment and workspace surfaces to expose sandbox environment configuration and selection. - Added test coverage for sandbox runtime behavior, provider seams, environment route guards, orchestration, and the fake provider plugin. ## Verification - Ran locally before the final fixture-only scrub: - `pnpm -r typecheck` - `pnpm test:run` - `pnpm build` - Ran locally after the final scrub amend: - `pnpm vitest run server/src/__tests__/runtime-api.test.ts` - Reviewer spot checks: - create a sandbox environment backed by the fake provider plugin - run work through that environment - confirm sandbox provider execution does not inherit host secrets implicitly ## Risks - This touches shared contracts, plugin SDK plumbing, server runtime orchestration, and UI environment/workspace flows, so regressions would likely show up as cross-layer mismatches rather than isolated type errors. - Runtime URL discovery and sandbox callback selection are sensitive to host/bind configuration; if that logic is wrong, sandbox-backed callbacks may fail even when execution succeeds. - The fake provider plugin is intentionally deterministic and test-oriented; future providers may expose capability gaps that this branch does not yet cover. ## Model Used - OpenAI Codex coding agent on a GPT-5-class backend in the Paperclip/Codex harness. Exact backend model ID is not exposed in-session. Tool-assisted workflow with shell execution, file editing, git history inspection, and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Devin Foley
|
e4995bbb1c |
Add SSH environment support (#4358)
## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies > - The environments subsystem already models execution environments, but before this branch there was no end-to-end SSH-backed runtime path for agents to actually run work against a remote box > - That meant agents could be configured around environment concepts without a reliable way to execute adapter sessions remotely, sync workspace state, and preserve run context across supported adapters > - We also need environment selection to participate in normal Paperclip control-plane behavior: agent defaults, project/issue selection, route validation, and environment probing > - Because this capability is still experimental, the UI surface should be easy to hide and easy to remove later without undoing the underlying implementation > - This pull request adds SSH environment execution support across the runtime, adapters, routes, schema, and tests, then puts the visible environment-management UI behind an experimental flag > - The benefit is that we can validate real SSH-backed agent execution now while keeping the user-facing controls safely gated until the feature is ready to come out of experimentation ## What Changed - Added SSH-backed execution target support in the shared adapter runtime, including remote workspace preparation, skill/runtime asset sync, remote session handling, and workspace restore behavior after runs. - Added SSH execution coverage for supported local adapters, plus remote execution tests across Claude, Codex, Cursor, Gemini, OpenCode, and Pi. - Added environment selection and environment-management backend support needed for SSH execution, including route/service work, validation, probing, and agent default environment persistence. - Added CLI support for SSH environment lab verification and updated related docs/tests. - Added the `enableEnvironments` experimental flag and gated the environment UI behind it on company settings, agent configuration, and project configuration surfaces. ## Verification - `pnpm exec vitest run packages/adapters/claude-local/src/server/execute.remote.test.ts packages/adapters/cursor-local/src/server/execute.remote.test.ts packages/adapters/gemini-local/src/server/execute.remote.test.ts packages/adapters/opencode-local/src/server/execute.remote.test.ts packages/adapters/pi-local/src/server/execute.remote.test.ts` - `pnpm exec vitest run server/src/__tests__/environment-routes.test.ts` - `pnpm exec vitest run server/src/__tests__/instance-settings-routes.test.ts` - `pnpm exec vitest run ui/src/lib/new-agent-hire-payload.test.ts ui/src/lib/new-agent-runtime-config.test.ts` - `pnpm -r typecheck` - `pnpm build` - Manual verification on a branch-local dev server: - enabled the experimental flag - created an SSH environment - created a Linux Claude agent using that environment - confirmed a run executed on the Linux box and synced workspace changes back ## Risks - Medium: this touches runtime execution flow across multiple adapters, so regressions would likely show up in remote session setup, workspace sync, or environment selection precedence. - The UI flag reduces exposure, but the underlying runtime and route changes are still substantial and rely on migration correctness. - The change set is broad across adapters, control-plane services, migrations, and UI gating, so review should pay close attention to environment-selection precedence and remote workspace lifecycle behavior. ## Model Used - OpenAI Codex via Paperclip's local Codex adapter, GPT-5-class coding model with tool use and code execution in the local repo workspace. The local adapter does not surface a more specific public model version string in this branch workflow. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Devin Foley
|
13551b2bac |
Add local environment lifecycle (#4297)
## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies. > - Every heartbeat run needs a concrete place where the agent's adapter process executes. > - Today that execution location is implicitly the local machine, which makes it hard to track, audit, and manage as a first-class runtime concern. > - The first step is to represent the current local execution path explicitly without changing how users experience agent runs. > - This pull request adds core Environment and Environment Lease records, then routes existing local heartbeat execution through a default `Local` environment. > - The benefit is that local runs remain behavior-preserving while the system now has durable environment identity, lease lifecycle tracking, and activity records for execution placement. ## What Changed - Added `environments` and `environment_leases` database tables, schema exports, and migration `0065_environments.sql`. - Added shared environment constants, TypeScript types, and validators for environment drivers, statuses, lease policies, lease statuses, and cleanup states. - Added `environmentService` for listing, reading, creating, updating, and ensuring company-scoped environments. - Added environment lease lifecycle operations for acquire, metadata update, single-lease release, and run-wide release. - Updated heartbeat execution to lazily ensure a company-scoped default `Local` environment before adapter execution. - Updated heartbeat execution to acquire an ephemeral local environment lease, write `paperclipEnvironment` into the run context snapshot, and release active leases during run finalization. - Added activity log events for environment lease acquisition and release. - Added tests for environment service behavior and the local heartbeat environment lifecycle. - Added a CI-follow-up heartbeat guard so deferred issue comment wakes are promoted before automatic missing-comment retries, with focused batching test coverage. ## Verification Local verification run for this branch: - `pnpm -r typecheck` - `pnpm build` - `pnpm exec vitest run server/src/__tests__/environment-service.test.ts server/src/__tests__/heartbeat-local-environment.test.ts --pool=forks` Additional reviewer/CI verification: - Confirm `pnpm-lock.yaml` is not modified. - Confirm `pnpm test:run` passes in CI. - Confirm `PAPERCLIP_E2E_SKIP_LLM=true pnpm run test:e2e` passes in CI. - Confirm a local heartbeat run creates one active `Local` environment when needed, records one lease for the run, releases the lease when the run finishes, and includes `paperclipEnvironment` in the run context snapshot. Screenshots: not applicable; this PR has no UI changes. ## Risks - Migration risk: introduces two new tables and a new migration journal entry. Review should verify company scoping, indexes, foreign keys, and enum defaults are correct. - Lifecycle risk: heartbeat finalization now releases environment leases in addition to existing runtime cleanup. A finalization bug could leave stale active leases or mark a failed run's lease incorrectly. - Behavior-preservation risk: local adapter execution should remain unchanged apart from environment bookkeeping. Review should pay attention to the heartbeat path around context snapshot updates and final cleanup ordering. - Activity volume risk: each heartbeat run now logs lease acquisition and release events, increasing activity log volume by two records per run. ## Model Used OpenAI GPT-5.4 via Codex CLI. Capabilities used: repository inspection, TypeScript implementation review, local test/build execution, and PR-description drafting. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (N/A: no UI changes) - [x] I have updated relevant documentation to reflect my changes (N/A: no user-facing docs or commands changed) - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge |
